1. technology and computing
2. programming languages
3. c and c++

Using The GNU GDB Debugger:
Table of Contents
Chapter 0: Administrata
1.
2.
3.
4.
5.
6.
7.
8.
9.
Current State Of This Document
Why Write This Tutorial?
Acknowledgements And Dedication
Authorship And Copyright
About The Exercises
Thanks
A Plug For The EFF
A Request For Help
Changelog (Updated December 20, 2013)
Chapter 1: Introduction
1.
2.
3.
4.
What Is A Debugger?
Why Not Use printf()?
What Is GDB?
Other Symbolic Debuggers
Debuggers
Front Ends
Chapter 2: Memory Layout and the Stack
1.
2.
3.
4.
Before You Debug
Virtual Memory
Memory Layout
Stack Frames And The Stack
Chapter 2.5: Preparing An Executable For Debugging
1. Before You Debug (Part II)
Theory: Symbol Tables
Practical: Using GCC Debug Switches
Chapter 2.8: Examining The Stack With GDB
1. Loading A Program And Setting A Breakpoint
2. The Backtrace Command
3. The Frame Command
Interlude: How To Debug Without A Debugger
1. Prologue
2. Debugging With Your Brain
Chapter 3: Initialization, Listing, And Running
1. Recap & Roadmap
2. Listing Source Code
Listing By Memory Address (advanced)
Setting The List Size
3. The .gdbinit File
4. Running A Program In GDB
5. Restarting A Program In GDB
Chapter 4: Breakpoint Basics
1. Introduction To Breakpoints
2. What Is a Breakpoint?
3. Breaking
Setting Basic Breakpoints
Breakpoint Numbers
Listing Breakpoints
Enabling, Disabling, and Ignoring Breakpoints
Removing Breakpoints
Chapter 4.5: Various Ways To Set GDB Breakpoints
1. Basic Methods
By Line Number
By Function Name
Relative To Current Line Of Execution
2. Advanced Methods
By Filename & Line Number
By Filename & Function Name
By Address
By Next Instruction
By Next Instruction
Conditional Breakpoints
3. Summary
Chapter 5: Inspecting And Changing Variables
1.
2.
3.
4.
Inspecting Variables
Inspecting Arrays And Structures
Advanced Inspection
Changing Variables
Chapter 5.5: Moving Around In The Sourcecode
1. Stepping Through Your Program
2. Finding Out Where You Are And Listing Source Code
Chapter 6: Debugging A Running Process
1. How To Attach GDB To An Executing Program
With Command Line Arguments
With The Attach Command
2. Processes Without Debugging Symbols
Chapter 7: Debugging Ncurses Programs
1.
2.
3.
4.
Ncurses
A Sample Ncurses Debug Session
Separating Input/Output
Debugging Ncurses Example
Chapter 8: Other Stuff
1.
2.
3.
4.
Official GDB Sources
Formats For This Document
Other GDB Tutorials
Kudos
Using The GNU GDB Debugger:
Administrata
Current State Of This Document
Note:The author of this material is Peter Jay Salzman. Some pages have been edited. See the next two paragraphs
for more details.
When I first republished this document, Peter Salzman's site was still down, as it had been since early 2009. That has now
changed. As of May 2011, his site seems to be up and running again, albeit intermittently. For this reason, I will continue to
maintain this version of the tutorial. If you wish to see the original content, you can do so here. Also note that the old
www3.sympatico.ca/rsquared/gdb/ site is no longer maintained, and should be deleted shortly.
For the most part, I have left the subject material intact. Chapters four through 5.5 are exceptions. The documents I retrieved from
the internet archive had notation in them by the author, indicating that there was a reconstruction process going on. Clearly this
was the case, given the overlap of material in chapters four and five. I split the material up, moving some into chapter four,
renamed chapter five, and added chapters 4.5, and 5.5. Note: there is not any new material there at this time, only a general
reorganizing to make the subject matter flow better. Any other changes I made were mostly cosmetic (Adding a little white-space
on the margins, touching up some table borders and so on). I also worked on the html and css source to ensure they were
compliant with current W3 standards for their doctypes. Pages which have what I consider to be more than minor edits, have
Peter's name as the author, and my name as the editor. Pages with minor editing bear only Peter's name as the author. In the
future, I hope to add to this document, though that will happen only if I can make the time to do so.
If you wish to contact me about anything regarding this document (see A Request for Help below for some ideas of how you can
help), please contact me.
Why Write This Tutorial?
This is one of the most comprehensive GDB tutorials on the Internet. It's more than you'd find in most books, which tend to discuss
GDB as a lightning-fast afterthought. I wrote this document because I couldn't find a good GDB tutorial. The only comprehensive
source of information about GDB is GNU's GDB User's Manual, but learning GDB from it is like learning a foreign language from
a dictionary.
I'll be using sample programs, and there will be links to the source code in each section that uses them, along with compilation
instructions. I urge you to download the code and follow along with the examples. Following along, doing it yourself as you read, is
really the best way to learn.
Acknowledgements And Dedication
This tutorial is sincerely and respectfully dedicated to Richard M. Stallman, the most important and under appreciated hero of the
Free Software movement.
I'm in a perpetual state of learning, and thanks goes to the following people who've helped me understand C and GDB:
Will Deutsch: For answering questions about GDB.
Mike Simons: For answering questions about GDB.
Paul Hinton: of Wolfram Research for convincing me to try this crazy thing called "GNU/Linux".
Jeff Newmiller: Who has yet to be stumped by any question I throw at him.
Norm Matloff: Who seems to know everything that I don't know (which is a LOT!)
Mark K. Kim: Who never tires of my questions and has an amazing ability to incorporate out-of-box thinking with formal
learning. A true hacker, good friend, and humble guy.
Authorship And Copyright
Peter Salzman originally released this document under the GNU GPL. Even on the latest copy of the document, the license
seems fairly free and liberal, and so I am going to keep it that way.
This entire tutorial is copyright © 2004 Peter Jay Salzman. Permission is granted to copy, distribute and/or modify it under the
terms of The GNU Free Documentation License. You can find a copy of this license at www.gnu.org/licenses/fdl-1.3
The canonical and most updated version of this document can be found at rsquared.sdf.org/gdb/.
About the Exercises
There are exercises at the end of most sections. The exercises are mandatory. These exercises are designed to both cover
topics I don't formally cover, and give you experience using your new-found skills.
There are topics I don't cover except for in the exercises. This isn't because I'm lazy. It's because I want you to think. Use your
noggin to begin understanding concepts in your own words, not in my words. I want you to develop intuition. The best debugging
tool is not GDB, and it certainly isn't printf(). The best debugging tool is your brain.
Thanks
The following people sent in corrections (remove the "ZZZ" in the email address):
Nick
Jason E. Siefken (from Oregon State University?) Can someone please put me in touch with him?
Jason E. Siefken (from Oregon State University?) Can someone please put me in touch with him?
Eric T. Stuebe
Jeff Terrell
Lawrence Poorman
Yi Yang
Aaron Mayerson
Doug Yoder
A Plug For The Electronic Frontier Foundation (EFF)
If you're not a member of the EFF, you must stop everything you're doing and become a member right this moment. 9/11 was a
horrible tragedy; I was in New York City at the time and witnessed the chaos with my own two eyes. I love my country, and am a
very proud United States citizen, but the steady erosion of our freedoms and civil liberty is another tragic casualty of the post 9/11
era. I'm very worried for my country.
The EFF is the most important defense we have in protecting our on-line and digital rights. If you have any interest in protecting
your civil liberties in a digital age that has gone out of balance, please read about their work. Consider becoming a member of
the EFF. Honestly, it's only the price of a pizza. Or the cost of two movie tickets plus popcorn.
A Request For Help
This tutorial took (takes?) more time than I care to admit. It's a tremendous job. If you found this tutorial to be at all useful, please
consider helping me maintain and actively develop it. There are many ways you can help. Pick one (then contact me) that suits
you or your talents (in no particular order):
Report spelling errors, technical errors, and broken links.
Email me questions. Tell me if something isn't clear.
Suggest additional topics for coverage.
Note: Since the work has been rearranged, there is the definite possibility of errors, or holes, in parts of the document.
Particularly chapters 4-5. Please don't hesitate to send me a note, and make me aware of any problems with the
material. - editor, December 24, 2010
Changelog
12/20/2013: Fixed a few links, and added some material to the Front Ends section. Updated PDF to current.
08/30/2013: Fixed a few links. Removed the G+ images.
05/14/2013: Corrected a typo. (Thanks Paul!) Updated PDF to current.
08/20/2012: Updated Current State Of This Document. Fixed some links. Updated PDF to current.
05/22/2011: Updated Current State Of This Document. Updated PDF to current.
05/07/2011: Fixed a typo/spelling error. Updated PDF to current.
05/05/2011: Added link to a comparison of GDB and DBX features.
05/04/2011: Added section Formats For This Document. Updated PDF to current.
04/30/2011: Separated the gdb frontends section into current and historical. Added links for Nemiver and MyGDB
frontends.
Using The GNU GDB Debugger:
Introduction
What Is A Debugger?
A debugger (or more accurately, symbolic debugger), is an application that runs your program, just like you can, when you type
the name of your program. The difference is, a debugger can step through your source code, line by line, executing each line only
when you want it to. You can even step through your program machine instruction by machine instruction (try that with
printf())! At any point, you can inspect and even change the value of any variable at run-time. If your program crashes, a
symbolic debugger tells you where and why the program crashed so you can deduce what went wrong. You can go through the
program and see what source code lines get executed and in what order.
Do you have an infinite loop? No problem! Use a debugger to step through the loop and see why your conditional fails to do what
you had expected. Did the program crash on a variable access? No problem! The debugger will tell you all sorts of information
about the variable you tried to access and the value you assigned (or perhaps didn't assign) to it. Is there a line in your code
which isn't executing? No problem! Use the debugger to see what gets executed, in what order, and why a particular line isn't
getting reached! Other than a compiler, the debugger is the most useful tool a programmer can use.
Why Not Use printf()?
Most people use the printf() debugging method. This is called adding "trace code" to your program. Simply put, they sprinkle
their code with printf() to view the value of variables at certain strategic points and also to examine the order of execution of
lines of source code.
There are a few reasons why this may not be the best way of doing things:
1. Sometimes you need a lot of printf()'s, and it can get tedious putting them in and taking them out. Inserting and deleting
superfluous code all the time is really distracting. It draws attention away from what you're doing. It's like trying to implement
a linked list while someone is talking to you about last night's Futurama episode.
2. A symbolic debugger can do an awful lot that printf() can't. You can do just about anything you can think of, including
changing the value of variables at run-time, halt the program temporarily, list source code, print the datatype of a variable or
struct that you don't recognize, jump to an arbitrary line of code, and much, much more.
3. You can use a symbolic debugger on a running process; you don't even have to kill the process! Try that with printf()!
4. You can use a symbolic debugger on a process that has already crashed and died without having to re-run the program.
You'll see the state the program was in at the time of death and can inspect all the variables.
5. A knowledge of GDB will increase your knowledge of programs, processes, memory and your language of choice.
You'll be able to find and fix your bugs faster using a symbolic debugger like GDB. However, this isn't to say that printf() has
no use in debugging. Sometimes it's the best way to go. However, for real code, a debugger can almost always get the job done
orders of magnitude faster and easier. Using a debugger is always more elegant, and if you don't care about elegance, you
should quit programming on Linux and start using Visual C++.
What Is GDB?
In the previous section I told you what a symbolic debugger is. There are actually MANY symbolic debuggers, and in the next
section I'll mention some of them. However, this tutorial is about one particular debugger which I use, called GDB.
GDB is a debugger which is part of the Free Software Foundation's GNU operating system. Its original author is Richard M.
Stallman (affectionately known as "RMS", one of the finest heroes of the free software movement), and has a long and impressive
list of contributors, including some interesting corporate sponsorship for support under various architectures. It's a wonderful
piece of software and outclasses nearly every other debugger I've seen, including commercial ones.
GDB can be used to debug C, C++, Objective-C, Fortran, Java and Assembly programs. There's partial support for Modula-2
and Pascal. It'll run on any architecture you can think of that supports Unix, so learning GDB on your home PC will give you the
power to debug code anywhere Unix can run!
Way back when, dbx was the canonical debugger people used on Unix systems. With the advent of GNU being the standard by
which all Unix systems are measured, GDB became the canonical debugger of the debugging world. As a result, even
commercial debuggers have a tendency to be command compatible (or even idea compatible) with GDB, so learning GDB will
enable you to use a whole slew of other debuggers. In short, if you learn GDB, you will be able to debug anything almost anywhere
with any debugger in the Unix world.
GDB's homepage is located at www.gnu.org/software/gdb/gdb.html. As of December 2010, the current release is version 7.2.
GDB is copyleft software (meaning that not only is GDB free software, but all publicly released derivatives and enhancements
people make to GDB must also be free) and is licensed under the GNU GPL
.
Other Symbolic Debuggers
This section documents other debuggers, both actively developed and long gone. I give a short history when the information is
available. For any additions (history, debuggers not listed here, other front ends, screenshots) that you would like to see, please
let me know.
Debuggers
The first debugger that I know of was called dbx, and like GDB, was command line driven. Oracle still offers the dbx
debugger as part of the Oracle Solaris Studio IDE. At any rate, the text UI of GDB was written to resemble dbx, although the
two debuggers are not completely compatible. (You can view a handy table comparing gdb and dbx commands here, and a
two debuggers are not completely compatible. (You can view a handy table comparing gdb and dbx commands here, and a
more extensive one here). Other symbolic debuggers were written so that their UI resembled dbx (or GDB) as well. For this
reason, you'll find many command line debuggers to be quite similar. If you learn to use GDB, you'll largely be able to
navigate through most other debuggers.
ups is another debugger originally developed by Mark Russell but is now updated by Rod Armstrong. It also comes with its
own theme song. Ups includes a C interpreter which allows you to add fragments of code simply by editing them into the
source window (the source file itself is not modified). Perversely, this lets you add debugging printf() calls without
recompiling, relinking or even restarting the target program. ups supports C, C++ and limited FORTRAN debugging on
SunOS, Solaris, Linux and FreeBSD. Screenshots: old, new.
The Portland Group sells an excellent high-quality GUI debugger named pgdbg. pgdbg specializes in debugging all kinds of
parallel code on many different kinds of clusters (distributed memory, SMP servers, etc). While pgdb is a very highpowered debugger, it's also expensive. Screenshot.
Front Ends
Current
Perhaps the most popular GDB front end is DDD, the Data Display Debugger which uses the Motif widget set. DDD has
some nice features: it can give you graphical representations of linked lists, ADT's and trees. In addition, DDD is a front
end to the Python, Java and Perl debuggers as well. I personally don't use DDD much, but I still appreciate it. DDD used to
be quite buggy. Over the years it has stopped crashing regularly(!) on me, but as of March 2003, still crashes on a blue
moon. In addition, the pop-up command tool definitely has "issues" with window managers that have multiple screens, like
Enlightenment.
Nemiver is "an on going effort to write an easy to use standalone C/C++ debugger that integrates well in the GNOME
environment." You can read about its features here.
kdbg is another nice front end for gdb. It displays variable values in a tree structure, and also allows the user to display the
assembly code in line with the source code. You can see some screenshots at the kdbg site.
Insight is not technically a front-end for GDB (It is a version of gdb with a full graphical user interface). It sports a very nice
layout, with various options for displaying the source code. You can see what it looks like on the screenshots page.
cgdb is a text based front end to gdb, using the curses library. In some ways it is similar to the TUI mode of gdb, only with
more features such as colour. See it here.
Pyclewn "allows using vim as a front end to a debugger." Screenshot.
MyGDB (Site language is Korean. You can view the page translated mostly to English here.) MyGDB is multiplatform; it
even runs on Microsoft Windows.
(IDEs)
Kdevelop "is a free, open source IDE (Integrated Development Environment) for Linux, Solaris, FreeBSD, Max OS X and
other Unix flavors."
Eclipse CDT (C/C++ Development Tooling): another full featured IDE which is based on Java.
NetBeans, like Eclipse, is an IDE based on Java.
Code::Blocks is a capable, free, IDE which uses gcc and gdb for its build environment.
CodeLite is "an open-source, cross platform IDE for the C/C++ programming languages."
Qt Creator is " a cross-platform IDE (integrated development environment) tailored to the needs of Qt developers."
BVRDE is "a fully integrated development environment for remote cross-platform compiling and debugging of UNIX and
LINUX console applications. BVRDE runs on a Windows platform, but compiles and debugs applications on UNIX systems
or any system that allow a remote Telnet or SSH connection. Screenshots.
SlickEdit Non-free IDE.
Affinic serves as a non-free gui for GDB (there is also a version for LLDB). Screenshots here.
WinGDB is "an extension for Visual Studio allowing to develop programs with GNU tools." Screenshots. Non-free.
Historical
tgdb is a Tcl/Tk front end for GDB first written in 1994 by a company named HighTec EDV-Systeme GmbH, in Germany. It
was shareware (asking price was $30). Development and support seems to have ended many years ago. It shouldn't be
confused with "trivial gdb" which is also called tgdb. Does anyone have a screenshot?
xdbx is a front end to dbx (see next entry) that was created by Po Cheung of Microelectronics and Computer Technology
Corporation (MCC) in March 10, 1989. It uses the old X Athena widget set (libxaw). It has its own license which is open
source but not copyleft. Development died a long, long time ago. Screenshot.
xxgdb is a front end to GDB that was created in December 1990 by Pierre Willard. It has its own license which is open
source but not copyleft. It's built from the source code for xdbx; basically, xxgdb is xdbx adapted to GDB instead of dbx.
xxgdb uses the old X Athena widget set (libxaw). It currently doesn't run on any system that uses unix98 posix TTYs.
Development died in 2002. It most likely doesn't work with current versions of GDB. Screenshot.
mxgdb is a Motif based front end for GDB written by Jim Tsillas back in January 3 1992. mxgdb is based on xxgdb: Jim
ported xxgdb from the Athena widget set to the Motif widget set (in turn, xxgdb was a GDB port of xdbx). It's licensed under
the GNU GPL and was last maintained (I think) by Robert Stockmann. It most likely doesn't work with current GDB versions.
Does anyone have a screenshot?
Using The GNU GDB Debugger:
Memory Layout And The Stack
What You Need To Know Before You Debug
To effectively learn how to use GDB, you must understand frames, which are also called stack frames because they're the frames
that comprise the stack. To learn about the stack, we need to learn about the memory layout of an executing program. The
discussion will mainly be theoretical, but to keep things interesting we'll conclude the chapter with an example of the stack and
stack frames using GDB.
The material learned in this chapter may seem rather theoretical, but it does serve a few very useful purposes:
1. Understanding the stack is absolutely necessary for using a symbolic debugger like GDB.
2. Knowing the memory layout of a process will help us understand what exactly a segmentation fault (or segfault) is, and why
they happen (or sometimes, more importantly) don't happen when they should. In brief, segfaults are the most common
immediate cause for a program to bomb.
3. A knowledge of a program's memory space can often allow us to figure out the location of well-hidden bugs without the use
of print() statements, a compiler or even GDB! In the next section, which is a guest written piece by one my friends,
Mark Kim, we'll see some real Sherlock Holmes style sleuthing. Mark homes in on a well hidden bug in somewhat lengthy
code. It only took him about 5 or 10 minutes, and all he did was look at the program and use his knowledge of how a
program's memory space works. It's really impressive!
Virtual Memory (VM)
Whenever a process is created, the kernel provides a chunk of physical memory which can be located anywhere at all. However,
through the magic of virtual memory (VM), the process believes it has all the memory on the computer. You might have heard
"virtual memory" in the context of using hard drive space as memory when RAM runs out. That's called virtual memory too, but is
largely unrelated to what we're talking about. The VM we're concerned with consists of the following principles:
1. Each process is given physical memory called the process's virtual memory space.
2. A process is unaware of the details of its physical memory (i.e. where it physically resides). All the process knows is how
big the chunk is and that its chunk begins at address 0.
3. Each process is unaware of any other chunks of VM belonging to other processes.
4. Even if the process did know about other chunks of VM, it's physically prevented from accessing that memory.
Each time a process wants to read or write to memory, its request must be translated from a VM address to a physical memory
address. Conversely, when the kernel needs to access the VM of a process, it must translate a physical memory address into a
VM address. There are two major issues with this:
1. Computers constantly access memory, so translations are very common; they must be lighting fast.
2. How can the OS ensure that a process doesn't trample on another process's VM?
The answer to both questions lies in the fact that the OS doesn't manage VM by itself; it gets help from the CPU. Many CPUs
contain a device called an MMU: a memory management unit. The MMU and the OS are jointly responsible for managing VM,
translating between virtual and physical addresses, enforcing permissions on which processes are allowed to access which
memory locations, and enforcing read/write permissions on sections of a VM space, even for the process that owns that space.
It used to be the case that Linux could only be ported to architectures that had an MMU (so Linux wouldn't run on, say, an x286).
However, in 1998, Linux was ported to the 68000 which had no MMU. This paved the way for embedded Linux and Linux on
devices such as the Palm Pilot.
Exercises
1. Read a short Wikipedia blurb on the MMU
2. Optional: If you want to know more about VM, here's a link. This is much more than you need to know.
Memory Layout
That's how VM works. For the most part, each process's VM space is laid out in a similar and predictable manner:
High Address
Args and env vars
Stack
|
V
Unused memory
^
|
Heap
Uninitialized Data Segment (bss)
Initialized Data Segment
Low Address
Text Segment
<-- Command line arguments and environment variables
<-- Initialized to zero by exec.
<-- Read from the program file by exec.
<-- Read from the program file by exec.
Text Segment: The text segment contains the actual code to be executed. It's usually sharable, so multiple instances of a
program can share the text segment to lower memory requirements. This segment is usually marked read-only so a
program can't modify its own instructions.
Initialized Data Segment: This segment contains global variables which are initialized by the programmer.
Uninitialized Data Segment: Also named "bss" (block started by symbol) which was an operator used by an old
assembler. This segment contains uninitialized global variables. All variables in this segment are initialized to 0 or NULL
assembler. This segment contains uninitialized global variables. All variables in this segment are initialized to 0 or NULL
pointers before the program begins to execute.
The stack: The stack is a collection of stack frames which will be described in the next section. When a new frame needs
to be added (as a result of a newly called function), the stack grows downward.
The heap: Most dynamic memory, whether requested via C's malloc() and friends or C++'s new is doled out to the
program from the heap. The C library also gets dynamic memory for its own personal workspace from the heap as well. As
more memory is requested "on the fly", the heap grows upward.
Given an object file or an executable, you can determine the size of each section (realize we're not talking about memory layout;
we're talking about a disk file that will eventually be resident in memory). Given hello_world-1.c, Makefile:
1
2
3
4
5
6
7
8
9
10
// hello_world-1.c
#include <stdio.h>
int main(void)
{
printf("hello world\n");
return 0;
}
compile it and link it separately with:
$ gcc -Wall -Wextra -c hello_world-1.c
$ gcc -o hello_world-1 hello_world-1.o
You can use the size command to list out the size of the various sections:
$ size hello_world-1 hello_world-1.o
text
data
bss
dec
hex
filename
916
256
4
1176
498
hello_world-1
48
0
0
48
30
hello_world-1.o
The data segment is the initialized and uninitialized segments combined. The dec and hex sections are the file size in decimal
and hexidecimal format respectively.
You can also get the size of the sections of the object file using "objdump -h" or "objdump -x".
$ objdump -h hello_world-1.o
hello_world-1.o:
file format elf32-i386
Sections:
Idx Name
0 .text
1
2
3
4
5
Size
VMA
LMA
File off Algn
00000023 00000000 00000000 00000034 2**2
CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
.data
00000000 00000000 00000000 00000058 2**2
CONTENTS, ALLOC, LOAD, DATA
.bss
00000000 00000000 00000000 00000058 2**2
ALLOC
.rodata
0000000d 00000000 00000000 00000058 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
.note.GNU-stack 00000000 00000000 00000000 00000065 2**0
CONTENTS, READONLY
.comment
0000001b 00000000 00000000 00000065 2**0
CONTENTS, READONLY
Exercises
1. The size command didn't list a stack or heap segment for hello_world or hello_world.o. Why do you think that is?
2. There are no global variables in hello_world-1.c. Give an explanation for why size reports that the data and bss segments
have zero length for the object file but non-zero length for the executable.
3. size and objdump report different sizes for the text segment. Can you guess where the discrepancy comes from? Hint:
How big is the discrepancy? See anything of that length in the source code?
4. Optional: Read this link about object file formats.
Stack Frames And The Stack
You just learned about the memory layout for a process. One section of this memory layout is called the stack, which is a
collection of stack frames. Each stack frame represents a function call. As functions are called, the number of stack frames
increases, and the stack grows. Conversely, as functions return to their caller, the number of stack frames decreases, and the
stack shrinks. In this section, we learn what a stack frame is. A very detailed explanation here, but we'll go over what's important
for our purposes.
A program is made up of one or more functions which interact by calling each other. Every time a function is called, an area of
memory is set aside, called a stack frame, for the new function call. This area of memory holds some crucial information, like:
1. Storage space for all the automatic variables for the newly called function.
2. The line number of the calling function to return to when the called function returns.
3. The arguments, or parameters, of the called function.
Each function call gets its own stack frame. Collectively, all the stack frames make up the call stack. We'll use hello_world-2.c for
the next example.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include <stdio.h>
void first_function(void);
void second_function(int);
int main(void)
{
printf("hello world\n");
first_function();
printf("goodbye goodbye\n");
return 0;
}
void first_function(void)
{
int imidate = 3;
char broiled = 'c';
void *where_prohibited = NULL;
second_function(imidate);
imidate = 10;
}
void second_function(int a)
{
int b = a;
}
When the program starts, there's one stack frame, belonging to main(). Since
main() has no automatic variables, no parameters, and no function to return to,
the stack frame is uninteresting. Here's what the stack looks like just before the call
to first_function() is made.
Frame for main()
Frame for main()
When the call to first_function() is made, unused stack memory is used to
create a frame for first_function(). It holds four things: storage space for an
int, a char, and a void *, and the line to return to within main(). Here's what the call
stack looks like right before the call to second_function() is made.
Frame for first_function()
Return to main(), line 9
Storage space for an int
Storage space for a char
Storage space for a void *
Frame for main()
When the call to second_function() is made, unused stack memory is used
to create a stack frame for second_function(). The frame holds 3 things:
storage space for an int and the current address of execution within
second_function(). Here's what the stack looks like right before
second_function() returns.
Frame for first_function():
Return to main(), line 9
Storage space for an int
Storage space for a char
Storage space for a void *
Frame for second_function():
Return to first_function(), line 22
Storage space for an int
Storage for the int parameter named a
Frame for main()
When second_function() returns, its frame is used to determine where to
return to (line 22 of first_function()), then deallocated and returned to
stack. Here's what the call stack looks like after second_function() returns:
When first_function() returns, its frame is used to determine where to
return to (line 9 of main()), then deallocated and returned to the stack. Here's
what the call stack looks like after first_function() return:
Frame for first_function():
Return to main(), line 9
Storage space for an int
Storage space for a char
Storage space for a void *
Frame for main()
And when main() returns, the program ends.
Exercises
1. Suppose a program makes 5 function calls. How many frames should be on the stack?
2. We saw that the stack grows linearly downward, and that when a function returns, the last frame on the stack is deallocated
and returned to unused memory. Is it possible for a frame somewhere in the middle of the stack to be returned to unused
memory? If it did, what would that mean about the running program?
3. Can a goto() statement cause frames in the middle of the stack to be deallocated? The answer is no, but why?
4. Can longjmp() cause frames in the middle of the stack to be deallocated?
Using The GNU GDB Debugger:
Preparing An Executable For Debugging
Before You Debug (Part II)
The first step of the debugging process is not debugging, but preparing the executable for debugging. In short, we need to add
information to the program. The next section briefly describes the reason for preparing an exectuable with an enhanced symbol
table. The final section describes how to prepare it by using the proper gcc debug switches.
Theory: Symbol Tables
A symbol is a variable or a function. A symbol table is exactly what you think: it's a table of variables and functions within an
executable. Normally, symbol tables contain only memory addresses of symbols, since computers don't use (or care) what we
name variables and functions.
But in order for GDB to be useful to us, it needs to be able to refer to variable and function names, not their addresses. Humans
use names like main() or i. Computers use addresses like 0x804b64d or 0xbffff784. To that end, we can compile code
with "debugging information" which tells GDB two things:
1. How to associate the address of a symbol with its name in the source code.
2. How to associate the address of a machine code with a line of source code.
A symbol table with this extra debugging information is called an augmented or enhanced symbol table. Because GCC and
GDB run on so many different platforms, there are many different formats for debugging information:
stabs: The format used by DBX on most BSD systems.
coff: The format used by SDB on most System V systems before System V Release 4.
xcoff: The format used by DBX on IBM RS/6000 systems.
dwarf: The format used by SDB on most System V Release 4 systems.
dwarf2: The format used by DBX on IRIX 6.
vms: The format used by DEBUG on VMS systems.
In addition to debugging formats, GDB understands enhanced variants of these formats that allow it to make use of GNU
extensions. Debugging an executable with a GNU enhanced debugging format with something other than GDB will can result in
anything from it working correctly to the debugger crashing.
Don't let all these formats scare you: in the next section, I'll show you that GDB automagically picks whatever format is best for
you. And for the 0.1% of you that need a different format, you're already knowledgeable enough to make that decision.
Practical: Using GCC Debug Switches
If you plan on debugging an executable, a corefile resulting from an executable, or a running process, you must compile the
executable with an enhanced symbol table. To generate an enhanced symbol table for an executable, we must compile it with
gcc's -g option:
gcc -g -o filename filename.c
As previously discussed, there are many different debugging formats. The actual meaning of -g is to produce debugging
information in the native format for your system.
As an alternative to -g, you can also use gcc's -ggdb option:
gcc -ggdb -o filename filename.c
which produces debugging information in the most expressive format available, including the GNU enhanced variants previously
discussed. I believe this is probably the option you want to use in most cases.
You can also give a numerical argument to -g, -ggdb and all the other debugging format options, with 1 being the least amount
of information and 3 being the most. Without a numerical argument, the debug level defaults to 2. By using -g3 you can even
access preprocessor macros, which is really nice. I suggest you always use -ggdb3 to produce an enhanced symbol table.
Debugging information compiled into an executable will not be read into memory unless GDB loads the executable. This means
that executables with debug information will not run any slower than executables without debug information (a common
misconception). While it's true that debugging executables take up more disk space, the executable will not have a larger
"memory footprint" unless it's from within GDB. Similarly, executable load time will be nearly the same, again, unless you run the
debug executable from within GDB.
One last comment. It's certainly possible to perform compiler optimizations on an executable which has an augmented symbol
table, in other words: gcc -g -O9 try1.c. In fact, GDB is one of the few symbolic debuggers which will generally do quite
well debugging optimized executables. However, you should generally turn off optimizations when debugging an executable
because there are situations that will confuse GDB. Variables may get optimized out of existence, functions may get inlined, and
more things may happen that may or may not confuse gdb. To be on the safe side, turn off optimization when you're debugging a
program.
Exercises
1. Using what you have learned in the last section, download the file try1.c, and compile it with debugging information.
2. Run "strip --only-keep-debug try1". Look at the file size of try1. Now run "strip --strip-debug try1
and look at the file size. Now run strip --strip-all try1 and look at the file size. Can you guess what's
happening? If not, your punishment is to read "man strip", which makes for some provocative reading.
happening? If not, your punishment is to read "man strip", which makes for some provocative reading.
3. You stripped all the unnecessary symbols from try1 in the previous exercise. Re-run the program to make sure it works.
Now run "strip --remove-section=.text try1" and look at the file length. Now try to run try1. What do you
suppose is going on?
4. Read this link about symbol tables (it's short).
5. Optional: Read this link about the COFF object file format.
Using The GNU GDB Debugger:
Examining The Stack With GDB
Loading A Program And Setting A Breakpoint
We'll look at the stack again, this time, using GDB. You may not understand all of this since you don't know about breakpoints yet,
but it should be intuitive. Compile and run try1.c:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#include<stdio.h>
static void display(int i, int *ptr);
int main(void) {
int x = 5;
int *xptr = &x;
printf("In main():\n");
printf("
x is %d and is stored at %p.\n", x, &x);
printf("
xptr points to %p which holds %d.\n", xptr, *xptr);
display(x, xptr);
return 0;
}
void display(int z, int *zptr) {
printf("In display():\n");
printf("
z is %d and is stored at %p.\n", z, &z);
printf("
zptr points to %p which holds %d.\n", zptr, *zptr);
}
Make sure you understand the output before continuing with this tutorial. Here's what I see:
$ ./try1
In main():
x is 5 and is stored at 0xbffff948.
xptr points to 0xbffff948 which holds 5.
In display():
z is 5 and is stored at 0xbffff924.
zptr points to 0xbffff948 which holds 5.
You debug an executable by invoking GDB with the name of the executable. Start a debugging session with try1. You'll see a
rather verbose copyright notice:
$ gdb try1
GNU gdb 6.1-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
(gdb)
The (gdb) is GDB's prompt. It's now waiting for us to input commands. The program is currently not running; to run it, type run.
This runs the program from inside GDB:
(gdb) run
Starting program: try1
In main():
x is 5 and is stored at 0xbffffb34.
xptr points to 0xbffffb34 which holds 5.
In display():
z is 5 and is stored at 0xbffffb10.
zptr points to 0xbffffb34 which holds 5.
Program exited normally.
(gdb)
Well, the program ran. It was a good start, but frankly, a little lackluster. We could've done the same thing by running the program
ourself. But one thing we can't do on our own is to pause the program in the middle of execution and take a look at the stack.
We'll do this next.
You get GDB to pause execution by using breakpoints. We'll cover breakpoints later, but for now, all you need to know is that
when you tell GDB break 5, the program will pause at line 5. You may ask: does the program execute line 5 (pause between 5
and 6) or does the program not execute line 5 (pause between 4 and 5)? The answer is that line 5 is not executed. Remember
these principles:
1. break 5 means to pause at line 5.
2. This means GDB pauses between lines 4 and 5. Line 4 has executed. Line 5 has not.
Set a breakpoint at line 10 and rerun the program:
(gdb) break 10
Breakpoint 1 at 0x8048445: file try1.c, line 10.
(gdb) run
Starting program: try1
Starting program: try1
In main():
x is 5 and is stored at 0xbffffb34.
xptr holds 0xbffffb34 and points to 5.
Breakpoint 1, main () at try1.c:10
10
display(x, xptr);
The Backtrace Command
We set a breakpoint at line 10 of file try1.c. GDB told us this line of code corresponds to memory address 0x8048445. We
reran the program and got the first 2 lines of output. We're in main(), sitting before line 10. We can look at the stack by using
GDB's backtrace command:
(gdb) backtrace
#0 main () at try1.c:10
(gdb)
The gdb backtrace command simply lists all of the frames currently on the stack. In the example above, there is one frame on the
stack, numbered 0, and it belongs to main(). If we execute the next line of code, we'll be in display(). From the previous
section, you should know exactly what should happen to the stack: another frame will be added to the bottom. Let's see this in
action. You can execute the next line of code using GDB's step command:
(gdb) step
display (z=5, zptr=0xbffffb34) at try1.c:15
15
printf("In display():\n");
(gdb)
Look at the stack again, and make sure you understand everything you see:
(gdb) backtrace
#0 display (z=5, zptr=0xbffffb34) at try1.c:15
#1 0x08048455 in main () at try1.c:10
Some points to note:
We now have two stack frames, frame 1 belonging to main() and frame 0 belong to display().
Each frame listing gives the arguments to that function. We see that main() took no arguments, but display() did (and
we're shown the value of the arguments).
Each frame listing gives the line number that's currently being executed within that frame. Look back at the source code and
verify you understand the line numbers shown in the backtrace.
Personally, I find the numbering system for the frame to be confusing. I'd prefer for main() to remain frame 0, and for
additional frames to get higher numbers. But this is consistent with the idea that the stack grows "downward". Just
remember that the lowest numbered frame is the one belonging to the most recently called function.
Execute the next two lines of code:
(gdb) step
In display():
16
printf("
z is %d and is stored at %p.\n", z, &z);
(gdb) step
z is 5 and is stored at 0xbffffb10.
17
printf("
zptr holds %p and points to %d.\n", zptr, *zptr);
The Frame Command
Recall that the frame is where automatic variables for the function are stored. Unless you tell it otherwise, GDB is always in the
context of the frame corresponding to the currently executing function. Since execution is currently in display(), GDB is in the
context of frame 0. We can ask GDB to tell us which frame its context is in by giving the frame command without arguments:
(gdb) frame
#0 display (z=5, zptr=0xbffffb34) at try1.c:17
17
printf("
zptr holds %p and points to %d.\n", zptr, *zptr);
I didn't tell you what the word "context" means; now I'll explain. Since GDB's context is in frame 0, we have access to all the local
variables in frame 0. Conversely, we don't have access to automatic variables in any other frame. Let's investigate this. GDB's
print command can be used to give us the value of any variable within the current frame. Since z and zptr are variables in
display(), and GDB is currently in the frame for display(), we should be able to print their values:
(gdb) print z
$1 = 5
(gdb) print zptr
$2 = (int *) 0xbffffb34
But we do not have access to automatic variables stored in other frames. Try to look at the variables in main(), which is frame
1:
(gdb) print x
No symbol "x" in current context.
(gdb) print xptr
No symbol "xptr" in current context.
Now for magic. We can tell GDB to switch from frame 0 to frame 1 using the frame command with the frame number as an
argument. This gives us access to the variables in frame 1. As you can guess, after switching frames, we won't have access to
variables stored in frame 0. Follow along:
(gdb) frame 1
#1 0x08048455 in main () at try1.c:10
10
display(x, xptr);
(gdb) print x
$5 = 5
(gdb) print xptr
$6 = (int *) 0xbffffb34
(gdb) print z
No symbol "z" in current context.
(gdb) print zptr
No symbol "zptr" in current context.
<--- switch to frame 1
<--- we have access to variables in frame 1
<--- we have access to variables in frame 1
<--- we don't have access to variables in frame 0
<--- we don't have access to variables in frame 0
By the way, one of the hardest things to get used to with GDB is seeing the program's output:
x is 5 and is stored at 0xbffffb34.
xptr holds 0xbffffb34 and points to 5.
intermixed with GDB's output:
Starting program: try1
In main():
...
Breakpoint 1, main () at try1.c:10
10
display(x, xptr);
intermixed with your input to GDB:
(gdb) run
intermixed with your input to the program (which would've been present had we called some kind of input function). This can get
confusing, but the more you use GDB, the more you get used to it. Things get tricky when the program does terminal handling
(e.g. ncurses or svga libraries), but there are always ways around it.
Exercises
1. Continuing from the previous example, switch back to display()'s frame. Verify that you have access to automatic
variables in display()'s frame, but not main()'s frame.
2. Figure out how to quit GDB on your own. Control-d works, but I want you to guess the command that quits GDB.
3. GDB has a help feature. If you type help foo, GDB will print a description of command foo. Enter GDB (don't give GDB
any arguments) and read the help blurb for all GDB commands we've used in this section.
4. Debug try1 again and set a breakpoint anywhere in display(), then run the program. Figure out how to display the stack
along with the values of every local variable for each frame at the same time. Hint: If you did the previous exercise, and read
each blurb, this should be easy.
Using The GNU GDB Debugger:
How To Debug Without A Debugger
Prologue
As of SDL 1.2.11, it appears that SDL_SetVideoMode() no longer generates SIGFPE when passed SDL_OPENGL. This
means you can use GDB to debug spinning_cube. However, this is still an excellent example of:
1. How to debug with your brain.
2. Why knowing theory, like the memory layout of a program, can be helpful when debugging.
Debugging With Your Brain
In the last section we looked at how a program is laid out in memory. Knowing this is not only useful for debugging with GDB, but
it's also useful for debugging without GDB. In this interlude, guest written by my close friend, Mark Kim, we'll see how.
Compile and run spinning_cube.tar.bz2. A spinning cube is displayed with images of Geordi (white) and Juliette (calico), me on a
New York City subway, and where I work.
However, when you press a key, some of the cube's textures mysteriously vanish. My first instinct was to use GDB to find the
problem, but I discovered that SDL programs that use OpenGL can't be debugged via GDB. Upon investigation, I found that when
you pass the flag SDL_OPENGL to the function SDL_SetVideoMode(), a SIGFPE is generated which terminates the
program. If you try to handle the SIGFPE, you'll find that SDL_SetVideoMode() never returns, so GDB is left in a hung state.
I had just spent over 40 hours programming over the last 3 days and was getting punch-drunk. Not having GDB available pushed
me over the edge and I sent an exasperated email to Mark for help. I got a reply within 10 minutes.
Before continuing you'll want to:
1. Run the program to see the bug in action. You need OpenGL and SDL to compile the program.
2. Look at HandleKeyPress() in input.c, which handles keystrokes.
3. Look at Debug(), in yerror.h, which is called from HandleKeyPress().
Spend 10 minutes trying to fix the bug. This will make Mark's email all the more impressive. As you read Mark's email, pay
particular attention to steps 6, 7B, and 7C for particular examples of sheer debugging brilliance!
Hey Peter,
The problem was there was an overlapping memory area between the debugging
variables and the texture variabes. In video.[hc], the "texture[2]" array
should have been declared "texture[NUM_TEXURES]" instead. Attached is a
patch file.
The debugging process went like this:
1. Try Debug() -- indeed it makes some textures disappear.
2. Try debug_for_reals() into an empty function -- same happens,
so that's not the problem.
3. Try removing each line of Debug() macro. This revealed that
writing values into the "die_*" variables cause the texture
to disappear.
4. So instead of calling Debug(), try writing some values into
the "die_*" variables -- the textures disappear again.
5. Check if any other code is using those variables by changing
variable names and looking out for compilation errors -nothing significant showed up.
6. Perhaps someone is using the same memory space as the "die_*"
variables unintentionally. I tried shifting the memory locations of
the "die_*" variables down by putting an array in front of them,
like this:
yerror.c:
...
#include "yerror.h"
+ char buffer[1024];
// Global Debugging/Dying Variables
const char *die_filename;
const char *die_function;
int
die_line;
bool
debug = true;
which fixed the problem.
overlapping memory.
So now it's a matter of finding the
overlapping memory.
7. Tracking down the problem needs some narrowing down of the
possiblilities, so I made the following assumptions:
A. I know a problem like this occurs most often when an array
size is declared too short at another place, so there's probably
an array out there that's declared too short, and the "die_*"
variables, placed in memory right after that array, is probably
getting overwritten by some code expecting the array to be
longer.
It could also be a pointer combined with malloc() but at this
point I'm just thinking about one problem at a time.
B. The problem must be with either a global or static variable
since it's overlapping with another global variable
in the heap space. So I'm looking for an array declared in
global or static scope. That narrows down my search quite a bit.
BTW, the fact that I'm looking for a variable that overlaps with
a global variable probably discounts malloc() from our potential
list of problems since malloc(), if the way I view the memory is
correct, should allocate memory only *after* all global
variables, and it's unlikely code accidentally writes to
a memory location before a pointer rather than an after
(though it's certainly possible to write to memory before
a pointer.) But again, this is all an afterthought... I'm just
thinking about another global array at this point.
C. I know the global array I'm looking for must be somehow linked
to a texture operation since that's what's being interfered by
writing to the "die_*" variables. So I'm looking for a global
array that does something with textures, probably one that stores
textures or pointers to textures or index to textures or
something like that.
8. And that's what I looked for. texture[2] looked a little suspicious
so I tried expanding its size and that fixed the problem. Just to
make sure, I looked for the code that writes to texture with index
greater than 1 and found init.c:127 and several places in render.c.
Hope that helps!
-Mark
Using The GNU GDB Debugger:
Initialization, Listing, And Running
Recap And Roadmap
In the last chapter we learned that the memory layout of an executing process is divided into segments. One important segment is
the call stack (or stack), which is a collection of stack frames (or frames). There is one frame for each function call, and the frame
holds three important things:
1. The local variables for the function.
2. The current address pointer within the function.
3. The arguments passed to the function.
When a function is called, a new frame is allocated and added to the stack. When the function returns, its frame is returned back
to unused stack memory and execution resumes at the address pointed to by the previous function's current address pointer. We
can ask GDB to tell us what the stack looks like with the backtrace command. We can also find out which frame GDB's context
is in using the frame command. We can also change GDB's context to the n'th frame using the frame n command.
Executables don't contain references to object (function and variable) names or source code line numbers. It would be painful to
debug a program without these things, so to debug a program, we generate an augmented symbol table using gcc's -g option.
Finally, we briefly considered how to make GDB pause execution using the break command and execute one line of source
code using the step command. We'll have much more to say about these commands shortly.
In this chapter, we'll investigate the list command which (surprisingly) lists lines of source code. We'll take an in-depth look at
GDB's initialization file .gdbinit. Lastly, we'll take a look at GDB's run command which executes a program from within GDB.
Basic Listing of Source Code
Download derivative, a program that calculates numerical derivatives, to follow along with the discussion: derivative.tar.bz2.
Take a moment to familiarize yourself with the code. Note the use of groovy function pointers.
You can list source code with GDB's list command, abbreviated by l. Run GDB on the executable and use the list
command:
$ gdb
(gdb)
12
13
14
15
16
17
18
19
20
21
driver
list
}
int main(int argc, char *argv[])
{
double x, dx, ans;
double Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta;
if (argc != 1) {
By default, GDB always lists 10 lines of source code. When you first issue list, GDB lists 10 lines of source code centred on
main(). Subsequent use of list gives the next 10 lines of source code. Try it:
(gdb)
22
23
24
25
26
27
28
29
30
31
(gdb)
list
printf("You must supply a value for the derivative location!\n");
return EXIT_FAILURE;
}
x
= atol(argv[1]);
ans = sin(log(x)) / x;
printf("%23s%10s%10s%11s%10s%11s\n", "Forward", "error", "Central",
"error", "Extrap", "error");
Use list three more times, and you'll see:
... output suppressed
45
printf("dx=%e: %.5e %.4f %.5e %.4f %.5e %.4f\n",
46
dx, Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta);
47
}
48
49
return 0;
50
}
(gdb) list
Line number 51 out of range; driver.c has 50 lines.
(gdb)
The second time we used list, only 9 lines were printed, since we reached the end of the file. The final list didn't print any
lines. That's because list always prints 10 lines of code after the previously listed lines. There were simply no more lines of
code to list.
"list -" works like list, except in reverse. It lists the 10 lines previous to the last listed lines. Since line 50 was the last listed
line, list -should print lines 41 through 50:
(gdb) list 41
42
Extr
= ExtrapolatedDiff(x, dx, &f);
43
ExtrDelta = fabs(Extr - ans);
44
45
printf("dx=%e: %.5e %.4f %.5e %.4f %.5e %.4f\n",
46
dx, Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta);
47
}
48
49
return 0;
50
}
(gdb)
If you give list a line number, GDB lists 10 lines centered on that line number:
(gdb) list 13
8
9
double f(double x)
10
{
11
return cos(log(x));
12
}
13
14
15
16
int main(int argc, char *argv[])
17
{
(gdb)
I'm going to suppress the output to conserve space, however I strongly encourage you to follow along with my examples by
performing the operations in GDB yourself. Try to imagine what the output looks like before you actually perform the operation.
Other listing operations you'll find useful:
By a line number
(gdb)
Ending with a line number
(gdb)
Range between two numbers: (gdb)
By function name:
(gdb)
By function in another file:
(gdb)
By filename and line number: (gdb)
By filename and function name: (gdb)
list
list
list
list
list
list
list
5,
,28
21,25
f
CentralDiff
derivative.c:12
derivative.c:ForwardDiff
list has a "memory" of what file was last used to print source code. We started out by listing lines from driver.c. We then
switched to derivative.c by telling GDB to list CentralDiff(). So now, list is in the "context" of derivative.c.
Therefore, if we use list by itself again, it'll list lines lines from derivative.c.
(gdb) list
11
}
12
13
14
15
double ExtrapolatedDiff( double x, double dx, double (*f)(double) )
16
{
17
double term1 = 8.0 * ( f(x + dx/4.0) - f(x - dx/4.0) );
18
double term2 = ( f(x + dx/2.0) - f(x - dx/2.0) );
19
20
return (term1 - term2) / (3.0*dx);
But what if we wanted to start listing lines from driver.c again? How do we go back to that file? We simply list anything that
lives in driver.c, like a function or line number. All these commands will reset list's command context from derivative.c back to
driver.c:
list
list
list
list
list
main
f
driver.c:main
driver.c:f
driver.c:20
And so forth. The rules aren't complicated; you'll get the hang of them after debugging a few multi-file programs.
Listing By Memory Address (advanced)
Every function begins at some memory address. You can find this address with the print function (which we'll cover later). For
instance, we'll find the address for main():
(gdb) print *main
$1 = {int (int, char **)} 0x8048647 <main>
(gdb)
So main() lives at 0x8048647. We can use list using memory locations as well; the syntax is very C'ish:
(gdb) list *0x8048647
0x8048647 is in main (driver.c:17).
12
}
13
14
15
16
int main(int argc, char *argv[])
17
{
18
double x, dx, ans;
19
double Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta;
20
21
if (argc != 1) {
(gdb)
It stands to reason that 0x8048690 is also somewhere inside of main(). Let's find out:
(gdb) list *0x8048690
0x8048690 is in main (driver.c:26).
21
if (argc != 1) {
22
printf("You must supply a value for the derivative location!\n");
23
return EXIT_FAILURE;
24
}
25
26
x
= atol(argv[1]);
27
ans = sin(log(x)) / x;
28
29
printf("%23s%10s%10s%11s%10s%11s\n", "Forward", "error", "Central",
30
"error", "Extrap", "error");
(gdb)
Exercises
1. Using list and print *, figure out how many machine instructions are used for this line of code:
18
19
double x, dx, ans;
double Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta;
Think about this for a second; you'll learn a bit about compilers and machine instructions.
Setting The List Size
GDB lists code in increments of 10 lines. Maybe that's too much. Or maybe that's too little. You can tell GDB to change the listing
size with the set command and listsize variable:
(gdb) set listsize 5
(gdb) list main
15
16
int main(int argc, char *argv[])
17
{
18
double x, dx, ans;
19
double Forw, ForwDelta, Cent, CentDelta, Extr, ExtrDelta;
(gdb)
Exercises
1. There's actually a lot of things you can set. Issue help set from GDB's prompt. I'm not expecting you to read it all---I just
want you to marvel at how big the list is!
The .gdbinit File
Upon startup, GDB reads and executes an initialization file named .gdbinit. It can contain any command (eg set and break),
and more. For example, "set listsize" and "set prompt" can go into .gdbinit. There are two locations where GDB will
look for this file (in order):
1. In your home directory
2. In the current directory
You can put commands to be executed for all your programming projects in $HOME/.gdbinit and project-specific commands in
$PWD/.gdbinit.
You can comment your .gdbinit files with bash's "#". And blank lines, of course, are ignored.
Exercises
1. When you invoke GDB, it prints a copyright notice. Using GDB's man page, figure out how to prevent GDB from printing this
notice. Using your shell's alias feature, make an alias for "gdb" that invokes GDB, but supresses the copyright notice. I use
this alias myself.
2. Figure out how to reset GDB's prompt from (gdb) to something that tickles your fancy. Google would be a great way of
figuring this out. GDB's help utility would also be useful (hint: you want to "set" the prompt to something else). Modify
figuring this out. GDB's help utility would also be useful (hint: you want to "set" the prompt to something else). Modify
.gdbinit so that GDB uses your chosen prompt on startup.
3. You can even use terminal escape codes to put color in your GDB prompt! If you don't know about terminal color escape
codes, you can read about them here. One caveat: You have to use the octal code \033 for the escape character. So for
example, bold blue would be \033[01;34m. And then don't forget to turn the blue off, otherwise everything will be blue. I'll
let you figure out how to do that yourself! Thanks to Jeff Terrell for pointing this out to me!
Running A Program In GDB
Let's properly introduce the run command. Download and compile arguments.tar.bz2.
The run command with no arguments runs your program without command line arguments. If you want to give the program
arguments, use the run command with whatever arguments you want to pass to the program:
$ gdb arguments
(gdb) run 1 2
Starting program: try2 1 2
Argument 0: arguments
Argument 1: 1
Argument 2: 2
Program exited normally.
(gdb)
Nothing could be simpler. From now on, whenever you use run again, it'll automatically use the arguments you just used (ie, "1
2"):
(gdb) run
Starting program: arguments 1 2
Argument 0: arguments
Argument 1: 1
Argument 2: 2
Program exited normally.
(gdb)
until you tell it to use different arguments:
(gdb) run testing one two three
Starting program: arguments testing one two three
Argument 0: testing
Argument 1: one
Argument 2: two
Argument 3: three
Program exited normally.
(gdb)
Suppose you want to run the program without command line arguments? How do you get run to stop automatically passing
them? There's a "set args" command. If you give this command without any parameters, run will no longer automatically pass
command line arguments to the program:
(gdb) set args
(gdb) run
Starting program: arguments
Argument 0: try2
Program exited normally.
(gdb)
If you do give an argument to set args, those arguments will be passed to the program the next time you use run, just as if you
had given those arguments directly to run.
There's one more use for set args. If intend on passing the same arguments to a program every time you begin a debugging
session, you can put it in your .gdbinit file. This will make run pass your arguments to the program without you having to specify
them every time you start GDB on a given project.
Restarting A Program In GDB
Sometimes you'll want to re-start a program in GDB from the beginning. One reason why you'd want to do this is if you find that
the breakpoint you set is too late in the program execution and you want to set the breakpoint earlier. There are three ways of
restarting a program in GDB.
1. Quit GDB and start over.
2. Use the kill command to stop the program, and run to restart it.
3. Use the GDB command run. GDB will tell you the program is already running and ask if you want to re-run the program
from the beginning.
The last two options will leave everything intact: breakpoints, watchpoints, commands, convenience variables, etc. However, if
you don't mind starting fresh with nothing saved from your previous debugging session, quitting GDB is certainly an option.
You might be wondering why there's a kill command when you can either quit GDB with quit or re-run the program with run.
The kill command seems kind of superfluous. There are some reasons why you'd use this command, and you can read about
them here. That said, I've never used kill myself.
Using The GNU GDB Debugger:
Breakpoint Basics
Introduction To Breakpoints
So far you know how to list source code and run a program from within gdb. But you already knew how to do that without gdb.
What else does gdb give us? To do anything really useful with gdb, you need to set breakpoints which temporarily pause your
program's execution so you can do useful debugging work like inspecting variables and watching the program's execution in an
atomic line-by-line fashion. This right here is the magic of a symbolic debugger.
Breakpoints come in three flavors:
1. A breakpoint stops your program whenever a particular point in the program is reached. We will discuss breakpoints
momentarily.
2. A watchpoint stops your program whenever the value of a variable or expression changes.
3. A catchpoint stops your program whenever a particular event occurs.
Note: We will primarily discuss breakpoints for now, and cover watchpoints and catchpoints in another article. What
you need to know is that watchpoints and catchpoints are a special form of breakpoints. As such, their use is quite
similar; if you can use breakpoints properly, you already know most of what you need to know in order to use the other
two.
What Is A Breakpoint?
A breakpoint stops your program whenever a particular place in the program is reached. Here are some examples of what a
breakpoint does:
Mr. Computer, won't you please stop when...
you reach line 420 of the current source code file?
you enter the function validateInput()?
you reach line 2718 of the file video.c?
All those requests have one thing in common: they ask gdb to stop based on reaching some location within the program. That's
what a breakpoint does. There are two things I'd like to mention before we start:
1. What does "stopping at line 5" mean?
When gdb stops at "line 5", this means that gdb is currently waiting "between" lines 4 and 5. Line 5 hasn't executed yet.
Keep this in mind! You can execute line 5 with the next command, but line 5 has not happened yet.
2. Why did gdb stop here?
Sometimes you may be surprised at where gdb stops. You may have specified a breakpoint at line 5 of the source code,
but gdb could stop at line 7, for instance. This can happen for 2 reasons. First, if you compile a program with optimization
set, some lines of source code may be optimized out of existence; they exist in your source code, but not in the executable.
Secondly, not every line of source code gets compiled into machine code instruction. See the section on "until" (FIXME:
when I write it). Consider the code below:
1
2
3
4
5
6
7
8
9
#include <stdio.h>
int main( void )
{
int i;
i = 3;
return 0;
}
Inserting a breakpoint at line X makes your program pause at line Y...
unoptimized code
Breakpoint at line
Program pauses at line
1--4, main()
4
5, 6
6
7, 8
8
9
9
optimized code
Breakpoint set at line
Program pauses at line
1--4, main()
4
5--9
9
Each breakpoint, watchpoint, and catchpoint you set is assigned a number starting with 1. You use this number to refer to that
breakpoint. To see the list of all breakpoints and watchpoints you've set, type info breakpoints (which can be abbreviated
by i b. I show a sample resulting output:
(gdb) info breakpoints
Num Type
Disp Enb Address
What
1
breakpoint
keep y
0x080483f6 in main at try5.c:4
breakpoint already hit 1 time
2
breakpoint
keep n
0x0804841a in display at try5.c:14
breakpoint already hit 1 time
3
breakpoint already hit 1 time
hw watchpoint keep y
i
According to the output, there are two breakpoints, one at line 4 and the other at line 14 of the source code. They are assigned to
numbers 1 and 2 respectively. There is also a watchpoint set: the program will halt whenever the variable i (local to display())
changes value.
In addition to being assigned a number, each breakpoint and watchpoint can be enabled or disabled. A program's execution
won't stop at a disabled breakpoint or watchpoint. By default, when you create a new breakpoint or watchpoint, it's enabled. To
disable the breakpoint or watchpoint assigned to number n, type:
disable n
To re-enable this breakpoint or watchpoint, type:
enable n
If you look at the sample output of info breakpoints above, you'll see that breakpoint 2 has been disabled.
Breaking
To help with the discussion, there's some broken code you can download and follow along with: fgets.c, main.c, and fgets.h.
Compile the code with:
$ gcc -c -ggdb3 -Wall -Wextra fgets.c main.c
$ gcc -o fgets fgets.o main.o
Note that the compiler generated a warning. That's because we used -Wall -Wextra which instructs gcc to tell us when it
sees what it thinks might be a common programming error. The best way to debug your program is to not put the bugs in the
program to begin with. You should always use these gcc bug finding options. Let me be blunt here, and I hope I don't offend
anyone. It's stupid not to use -Wall -Wextra when you compile code. Plain and simple. Stupid. With a capital S. Most people
don't use them, even people who are clearly better programmers than me. That's because even smart people can do dumb
things. Don't you be dumb. Always use -Wall -Wextra.
The program is a password guessing program. Take a moment to look through the code to see how it works. The program is
ultra-simple so we can focus on learning GDB rather than trying to figure out complicated code like linked lists and whatnot. You
should be able to deduce how the program works (and what the password is) in under a few seconds. Now run the code and
notice it simply doesn't work. We'll first concentrate on learning how to set breakpoints, and then we'll debug the program.
Setting Basic Breakpoints
One simple way to set breakpoints is with a line number. The line number refers to the file GDB is currently in. Right now, we're in
main.c, so line numbers are with respect to that file for now. Let's set a breakpoint at line 9, where the printf() statement is.
(gdb) break 9
Breakpoint 2 at 0x804846b: file main.c, line 9.
(gdb)
GDB has a continue command which we haven't seen yet. Once GDB pauses due to a breakpoint, the continue command
will resume execution. Use continue to make sure that GDB pauses at line 9:
(gdb) continue
Continuing.
Breakpoint 2, main () at main.c:9
9
printf("I'm thinking of a word.
(gdb)
Let's see if you can guess it.\n");
Breakpoint Numbers
You might have noticed that each breakpoint is given an integer identifier. For example, we've set 4 breakpoints already, and the
last one we set (by address) was assigned the number 4. If you haven't noticed this, go back and take a look. Breakpoint
numbers are valuable to you, because various operations can be performed on a breakpoint, such as removing them. In order to
operate on a breakpoint, you have to be able to reference it, and the breakpoint number is the reference, or identifier of the
breakpoint.
Listing Breakpoints
So far, we've seen two commands that take a breakpoint's identifier as an argument: enable, and disable. (You can also
delete, but more about that in the next section.) There are many other commands as well, which we'll cover later; the point is,
that in conjuction with commands, breakpoint identifiers are indispensable, and you'll find yourself using them quite a bit. But how
do you remember the identifiers for your breakpoints, or even where your breakpoints were set to begin with? Use info
breakpoints to list all your breakpoints, their identifiers, and lots more information. If you still have GDB from the previous
subsection, try it out:
(gdb) info breakpoints
Num Type
Disp Enb Address
1
breakpoint
keep y
0x08048464
breakpoint already hit 1 time
2
breakpoint
keep y
0x0804846b
breakpoint already hit 1 time
3
breakpoint
keep y
0x08048477
What
in main at main.c:6
in main at main.c:9
in main at main.c:12
This is a very important command, and I find myself using it all the time. It should be completely self explanatory except for a
This is a very important command, and I find myself using it all the time. It should be completely self explanatory except for a
couple of things:
1. The Num field gives the identifier.
2. The Type field gives the type of breakpoint. There are different types of breakpoints, like hardware watchpoints, which we'll
cover shortly.
3. The Disp field (short for disposition) describes what will happen to the breakpoint the next time it's activated (the next time
it pauses execution). keep indicates nothing will happen to the breakpoint, however, it's possible to disable or even
remove a breakpoint the next time it's reached. These situations are identified by the Disp field.
Enabling, Disabling, And Ignoring Breakpoints
Once set, there are only two ways to get rid of a breakpoint: remove it or quit GDB. GDB will continually break at the breakpoint.
However, you'll sometimes find it useful to temporarily disable a breakpoint, that is, you do not want GDB to break at the
breakpoint, but you want to keep the breakpoint there in case you need to debug that section of code again.
Breakpoints can be enabled and disabled. Simply put, your program will pause at an enabled breakpoint, but it will not pause at a
disabled breakpoint.
You can enable or disable breakpoints using the enable and disable commands which take an argument of the breakpoint
identifier for the breakpoint you want to enable or disable. Let's take a look at this using the fgets program that we previously
used. Start a debugging session of fgets and place two breakpoints at lines 6, 9, and 12 of main.c:
$ gdb fgets
(gdb) break 6
Breakpoint 1 at 0x8048464: file main.c, line 6.
(gdb) break 9
Breakpoint 2 at 0x804846b: file main.c, line 9.
(gdb) break 12
Breakpoint 3 at 0x8048477: file main.c, line 12.
Disable breakpoint 2, run the program, and use continue to verify that breakpoint 2 does not pause execution.
(gdb) disable 2
(gdb) run
Starting program: code/fgets/fgets
Breakpoint 1, main () at main.c:6
6
char *word = "password";
(gdb) continue
Continuing.
I'm thinking of a word. Let's see if you can guess it.
Breakpoint 3, main () at main.c:12
12
while ( KeepGoing )
Confirmed, breakpoint 2 is disabled. Finally, enable breakpoint 2 and rerun the program. Use continue to verify that
breakpoint 2 now pauses execution:
(gdb) enable 2
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /www/p/linux/gdb/code/fgets/fgets
Breakpoint 1, main () at main.c:6
6
char *word = "password";
(gdb) continue
Continuing.
Breakpoint 2, main () at main.c:9
9
printf("I'm thinking of a word.
Let's see if you can guess it.\n");
Confirmed, once enabled, breakpoint 2 again pauses execution.
Exercises
1. The disable command permanently disabled a breakpoint until you explicitly enable it with enable. However, it's
possible to temporarily disable a breakpoint. Use GDB's help utility to read about the ignore command, which disables a
breakpoint "for n crossings".
2. Personally, I don't use ignore a whole lot. It seems like conditional breaking makes ignore not very useful, but you should
still know of its existence. Hopefully you have GDB still open. Use ignore to disable breakpoint 3 (the one at line 12) for 3
crossings. Verify that it works.
Removing Breakpoints
Just as you can set breakpoints, you can also remove them. There are numerous ways to remove a breakpoint:
If you want to remove the breakpoint by its location, use clear.
If you want to remove the breakpoint by its identifier, use delete.
So let's use clear to remove the four breakpoints the way we set them; kind of like "undoing" what we did:
(gdb) clear *0x80483f4
Deleted breakpoint 4
(gdb) clear fgets.c:10
Deleted breakpoint 3
(gdb) clear 9
Deleted breakpoint 2
(gdb) clear main
Deleted breakpoint 1
(gdb)
The delete command deletes breakpoints by identifier, as opposed to clear which removes breakpoints based on their
location. In fact, delete n deletes the breakpoint with identifier n. We investigate this command more fully in the exercises.
Here are the commands used to delete breakpoints in a tabular format:
clear <function>
clear <filename><function>
clear <linenum>
clear <filename:linenum>
delete
delete n
Clear any breakpoints set at the entry to the function <function>.
Clear any breakpoints set at the entry to the function <function> defined in the source code file
<filename>.
Clear any breakpoints set at line <linenum> of the current source file. The current source file is the last
file whose text was printed.
Clear any breakpoints at line <linenum> in file <filename>.
Clear all breakpoints.
Each breakpoint is assigned a number starting with 1. This clears breakpoint n.
Exercises
1. If you've been following along with the tutorial, you shouldn't have any breakpoints set since we deleted them all with clear.
Set three breakpoints wherever you like by the methods of your choice. Before you do, guess what their identifiers will be.
2. Use delete, not clear, to remove only the last breakpoint you set. This will leave you with two remaining breakpoints.
3. You should have two breakpoints left. delete with no arguments removes all breakpoints. Try it out, then quit GDB.
Using The GNU GDB Debugger:
Various Ways To Set GDB Breakpoints
Setting Breakpoints: Basic Methods
There are many ways to set breakpoints. We'll go over each in turn. Download try5.c and follow my example. First, compile try5.c
for debugging.
$ gcc -Wall -Wextra -ggdb3 -o try5 try5.c
By Line Number
The first (and easiest) way you can set a breakpoint is by specifying a linenumber. To break at line 6, simply type break 6.
$ gdb try5
(gdb) break 6
Breakpoint 1 at 0x80483f6: file try5.c, line 6.
By Function Name
You can also set breakpoints with a function name:
(gdb) break display
Breakpoint 2 at 0x804841a: file try5.c, line 15.
Disable the 1st breakpoint, and then look at what you've done:
(gdb) disable 1
(gdb) info breakpoints
Num Type
Disp Enb Address
What
1
breakpoint
keep n
0x080483f6 in main at try5.c:6
2
breakpoint
keep y
0x0804841a in display at try5.c:15
Now run the program. Remember, breakpoint 1 is disabled, so it'll stop at line 15.
(gdb) run
Starting program: /www/p/linux/gdb/try5
Breakpoint 2, display (x=3) at try5.c:15
15
for (i=0; i<x; ++i) {
(gdb)
Relative To Current Line Of Execution
We've seen 2 ways to set a breakpoint. Now here's a third. To set a breakpoint 2 lines down from the current line, use break
+2. Similarly, you can set a breakpoint 3 lines up from the current line by break -3. Let's set a breakpoint at line 18 and
continue the execution.
(gdb) break +3
Breakpoint 3 at 0x8048450: file try5.c, line 18.
(gdb) continue
Continuing.
i is 0.
i is 1.
i is 2.
Breakpoint 3, display (x=5) at try5.c:18
18
}
(gdb)
Go ahead and quit gdb to prepare for the next section.
Setting Breakpoints: Advanced Methods
Up to this point, we have only covered how to do breakpoints with single source file programs. Now we will consider multi-source
file programs, and how to set breakpoints across files.
By Filename And Line Number
For the form break linenumber, there is an ambiguity when you have a multiple file program. The line number of which file?
By default, the line number entered with the break command will correspond to whatever file contains the main() function. That
certainly is a reasonable default! But what if we wanted to break on, say, line 5 of a different file? This gives a fourth form for the
break command:
break filename:linenumber
This command will break on line linenumber of the source code file named filename. For example, break MyFuncs.c:102
will break on line 102 of the source code file MyFuncs.c. The fifth form is similar:
By Filename And Function Name
break filename:function
For example, break MyFuncs.c:MyPrintFunction. But unless you're using overloaded function names (you've defined a
function multiple times), this is superfluous since you're not allowed (in C) to have 2 definitions belonging to the same function
name.
By Address
If you're trying to debug a program that doesn't have debugging info compiled into the executable, you can't set breakpoints by
line number or function name. Instead you have to specify where to break by giving a memory address. This gives us our sixth
form:
break *address
By Next Instruction
The break command without any argument gives a seventh form (only one more to go). It sets a break point at the very next
instruction. Look at try5 again (having one eye on the source code will help here).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ gdb try5
(gdb) break display
Breakpoint 1 at 0x804841a: file try5.c, line 15.
(gdb) run
Starting program: /www/p/linux/gdb/try5
Breakpoint 1, display (x=5) at try5.c:15
15
for (i=0; i<x; ++i) {
(gdb) next
16
printf("i is %d.\n", i);
(gdb) print i
$1 = 0
(gdb) break
Breakpoint 2 at 0x8048430: file try5.c, line 16.
(gdb) continue
Continuing.
i is 0.
Breakpoint 2, display (x=5) at try5.c:16
16
printf("i is %d.\n", i);
(gdb) print i
$2 = 1
The astute reader will wonder why, on line 22, i has the value of 1 and not 0. We set the breakpoint on line 13 when i had the
value of 0. But the very next instruction (which is where we set the breakpoint) was just a printf statement (source code line
16). How in blazes did the printf increment the value of i?
Here's the answer. Once gdb stops at a breakpoint, it will ignore all other breakpoints until one line of instruction has executed.
Why does it do this? If this weren't the case, everytime you stopped at a breakpoint, you'd have to disable that breakpoint to
resume execution--you wouldn't be able to get past that breakpoint! If this doesn't make sense to you, think about it for awhile. If
you still can't get it, don't worry. It's a minor point.
There's one more use for breakpoint form seven, i.e., the break command with no arguments: If you change to a higher frame,
use break and then continue, the b
$ gdb try5
(gdb) break display
Breakpoint 1 at 0x804841a: file try5.c, line 15.
15
for (i=0; i<x; ++i) {
(gdb) backtrace
#0 display (x=3) at try5.c:15
#1 0x8048409 in main () at try5.c:8
#2 0x4003e46b in __libc_start_main () from /lib/libc.so.6
(gdb) frame 1
#1 0x8048409 in main () at try5.c:8
8
display(x);
(gdb) break
Breakpoint 2 at 0x8048409: file try5.c, line 8.
(gdb) continue
Continuing.
i is 0.
i is 1.
i is 2.
Breakpoint 2, 0x8048409 in main () at try5.c:8
8
display(x);
(gdb)
Can you see what happened here? We stopped at the top of display(), frame 0. We then switched to the frame 1 (main())
and issued the break command. This set a breakpoint at the very next instruction after the call to display(). We then
continued execution, and the program ran until it hit the very next instruction after display(). In essence, we set the breakpoint
so that execution would halt after display() returned.
To reiterate, the seventh form of breakpoint is used for loops when you're in the top most frame and returns from functions when
To reiterate, the seventh form of breakpoint is used for loops when you're in the top most frame and returns from functions when
you're not in the top most frame. Frankly, I don't find this terribly useful. When in a loop, I think the break +offset or break
linenumber is more convenient. For returning from functions, I find the finish command more useful (which might even have
it's own entry in this tutorial some day).
Conditional Breakpoints
The eighth, and last, form of break command is the conditional breakpoint. They are quite useful but little understood. Perhaps
part of the reason is that the gdb User Manual does a really poor job explaining them. Here is the form:
break ... if cond
where ... represents any one of the previous seven forms of breakpoints we've learned about already and cond is any
conditional in the language you're using. Here is an example:
$ gdb try5
(gdb) break 16 if i==2
Breakpoint 1 at 0x8048430: file try5.c, line 16.
(gdb) r
Starting program: /www/p/linux/gdb/try5
i is 0.
i is 1.
Breakpoint 1, display (x=3) at try5.c:16
16
printf("i is %d.\n", i);
We used the first form of break with the conditional i==2. We could've also used a test for inequality, like i!=2 or i>2. This is mega
useful when you're inside of a loop that's going to repeat a million times. This last form of break is your friend!
Summary Of Breakpoints
1:
2:
3:
4:
5:
Form
break line number
break function
break +/- number
break filename:line number
break filename:function name
6:
break *address
7:
break no arguments
8:
break ... if condition
Explanation
Set a breakpoint at line number linenumber
Set a breakpoint at function function.
Set a breakpoint number lines before or after current line of execution
Set a breakpoint at line linenum in source file filename.
Set a breakpoint at function function name in source file filename.
Set a breakpoint at address <address>. Use this to set breakpoints in parts of a program
that doesn't have debugging information or source files.
Set a breakpoint at the next instruction.
Set a breakpoint where condition is any conditional in the language being debugged, and
... is any one of the previous seven forms of setting a breakpoint.
Using The GNU GDB Debugger:
Inspecting And Changing Variables
Inspecting Variables
Note to Fortran users: All Fortran variables must be in lowercase, regardless of how they were capitalized in your
source code. This is because the Fortran standard specifies case independence when it comes to variables. Yes,
variable 'C' is variable 'c' in the Fortran standard. There are compilers out there that allow you to use case dependent
variables, but this is non-standard, and gcc mandates all lowercase variables. This was done to support legacy code.
The whole purpose of setting a breakpoint or watchpoint is to see what's going on with your variables, so let's take a look at
inspecting your variables. You can print the data type of a variable using the ptype command. Here are some examples:
(gdb) ptype argc
type = int
(gdb) ptype myfloat
type = float
(gdb) ptype argv
type = char **
(gdb) ptype mystring
type = unsigned char *
(gdb) pt myIntArray
type = int [10]
You can even use ptype to look at structures. Take, for example, the fstat structure defined in sys/stat.h.
(gdb) ptype fstat
type = struct stat {
__dev_t st_dev;
short unsigned int __pad1;
__ino_t st_ino;
__mode_t st_mode;
__nlink_t st_nlink;
__uid_t st_uid;
__gid_t st_gid;
__dev_t st_rdev;
short unsigned int __pad2;
__off_t st_size;
long unsigned int st_blksize;
__blkcnt_t st_blocks;
__time_t st_atime;
long unsigned int __unused1;
__time_t st_mtime;
long unsigned int __unused2;
__time_t st_ctime;
long unsigned int __unused3;
long unsigned int __unused4;
long unsigned int __unused5;
}
That's quite a structure! You can abbreviate ptype by pt.
(gdb) pt mydouble
type = double
Remember, you can only print the data type of a variable which is defined in the currently selected frame.
Now that you know how to print the data type of your variables, you may want to print their values. Consider the following program
(which will be compiled via gcc -g filename):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include<stdio.h>
#include<string.h>
int main( int argc, char *argv[] )
{
double mydouble = 3.14 / 3;
float myfloat = 3.3;
char
mychar
= 'A';
int
myIntArray[10];
int
MyNegativeInt = -1;
char
myString[20];
struct foo {
char *name;
int EyeColour;
} myStruct;
strncpy(myString, "hello", 19);
for ( int i = 0; i < 10; i++ )
20
21
22
23
24
for ( int i = 0; i < 10; i++ )
myIntArray[i] = i;
return 0;
}
You can view the value of a variable using the print command.
(gdb) print i
$4 = -1073744780
I stopped the program right before the for loop, so this is what variable i is before it gets initialized. gdb prints the value of the
variable which is most `comfortable' (to borrow fortran 99 lingo) with the datatype. In other words, floats get printed as floats:
(gdb) print myfloat
$1 = 3.29999995
and doubles get printed as doubles:
(gdb) print mydouble
$1 = 1.0466666666666666
and chars get printed as chars:
(gdb) print mychar
$1 = 65 'A'
By the way, you can use the abbreviation p for print:
(gdb) p argc
$1 = 1
You may be wondering what the numbers preceeded by $ (like $1 or $3) mean. They're kind of like a variable history. Everytime
you print any variable, the $n gets incremented by 1. $ by itself refers to the last variable you printed and $n refers to the n'th
variable you printed. Look at the following example to see this:
(gdb)
$26 =
(gdb)
$27 =
(gdb)
$28 =
(gdb)
$29 =
(gdb)
$30 =
p mychar
65 'A'
p mydouble
1.0466666666666666
p $
1.0466666666666666
p $27
1.0466666666666666
p $26
65 'A'
You can even typecast a variable when you print it! Here's MyNegativeInt as an int, char and double respectively:
(gdb)
$41 =
(gdb)
$42 =
(gdb)
$43 =
p MyNegativeInt
-1
p (char) MyNegativeInt
-1 '� '
p (double) MyNegativeInt
-1
The possibilities are endless. But wait, there's more!
Inspecting Arrays And Structures
Printing array values is much the same as printing other variables. gdb still uses the concept of being `comfortable'. In other
words, when you print an array, that's exactly what you get! From the code snippet of the previous section:
(gdb) p myIntArray
$46 = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
Of course, gdb knows how to access elements of an array:
(gdb) pt myIntArray
type = int [10]
(gdb) pt myIntArray[3]
type = int
(gdb) p myIntArray[3]
$48 = 3
You can do kind of advanced stuff too -- things that you'd expect from only Perl :-). Here's how you print 5 elements of myIntArray,
starting at element 3:
(gdb) p myIntArray[3]@5
$49 = {3, 4, 5, 6, 7}
GDB will not, however, check bounds of the array. Previously we defined myIntArray as an array of 10 ints. Let's see what
happens when we try printing 4 ints past the end of the array:
(gdb) p myIntArray[3]@11
(gdb) p myIntArray[3]@11
$54 = {3, 4, 5, 6, 7, 8, 9, 10, 1107293224, 1079194419, -1947051841}
Doh! Hopefully, that's not someone's password. :-). You can also print structures:
(gdb) p myStruct
$2 = {name = 0x40014978 "Miles Davis", EyeColour = 1}
However, this might get out of hand for very large structs. You can set pretty printing of structures by set print pretty:
(gdb) set print pretty
(gdb) p myStruct
$4 = {
name = 0x40014978 "Miles Davis",
EyeColour = 1
}
(gdb)
or, if you only want one of the elements of the structure, you can print it in the way that would seem obvious:
(gdb) print myStruct.name
$6 = 0x40014978 "Miles Davis"
this works too, but why is a mystery to me:
(gdb) print myStruct->name
$15 = 0x40014978 "Miles Davis"
Advanced Inspection
You can print things using a format specifier:
print /FMT variable
Where FMT is:
o
t
octal
binary
x
f
hex
float
d
a
decimal
address
u
c
unsigned decimal
char
Here's some examples of printing some of our variables using a format specifier:
(gdb)
$33 =
(gdb)
$34 =
(gdb)
$35 =
(gdb)
$36 =
(gdb)
$37 =
(gdb)
$38 =
(gdb)
$39 =
(gdb)
$40 =
p mychar
65 'A'
p /o mychar
0101
p /x mychar
0x41
p /d mychar
65
p /u mychar
65
p /t mychar
1000001
p /f mychar
65
p /a mychar
0x41
By the way, memory addresses in gdb are printed in hex by default. Therefore, p /a mychar prints mychar interpreted as an
address, the hexidecimal representation of 65. This is very different from the address of mychar!
Speaking of the address of mychar, one would expect that since C loves pointers, gdb would love pointers too. And in fact, it
does! Printing the address of mychar is obvious to C programmers (sorry, Fortran users!):
(gdb) p &mychar
$42 = 0xbffff41b "A33S@� X� \213%�� ?H��� \023� \003@\001"
gdb even knows about the dereference operator. How's this for being perverse?
(gdb) p *(&mychar)
$43 = 65 'A'
This is the perfect vehicle for teaching students what a pointer is. We're dereferencing the address of mychar. Of course, there's
more to this than just coolness (although it's worth it for the coolness factor alone!). I was writing a curses program once and it
kept segfaulting on me whenever I tried drawing to a WINDOW object. By looking at the address of a WINDOW that I was
passing to a function, I determined that I was passing a WINDOW by value, drawing to a local copy of the WINDOW and returning.
Of course, the local copy of the WINDOW wasn't anything initialized by curses so drawing to it was causing a segmentation
violation. Looking at the code, it was highly non-obvious what was was going on; it looked just swell! It wasn't until I compared the
address of the passed WINDOW with the address of the received WINDOW that I discovered the big oops!
Furthermore, who here is guilty of buffer overruns? Be truthful! It's very easy to fall into the `off by one' error when you initialize,
write to or read from a C array. How many times have you used strcpy when you should've used strncpy? These errors are
insidious because they usually don't crash the program, but manifest themselves in wierd behavior in certain rare cases that are
hard to track down. Looking at the addresses of what's going on is a sure fire way of finding out the details of what's going on.
Changing Variables
There are two ways you can change the value of a variable in gdb. Let's change the value of double myvariable to 10.0.
Firstly, you can use the set command:
set myvariable = 10.0
which is the `quiet' way. gdb will simply set myvariable to 10 without printing anything. Then there's the `noisy' way using the
print command:
print myvariable = 10.0
which will set myvariable to 10.0 and then print this new value to the screen. The print command ends up being less
keystrokes because you can use the abbreviation p for print.
Remember, you can only change the value of a variable which is defined within the current context. Make sure the variable you
want to change is defined in the currently selected frame. If it's not, you need to set the frame before you can change the variable.
Using The GNU GDB Debugger:
Moving Around In The Sourcecode
Stepping through your program
One thing that is good to know is the exact sequence of execution of your program, especially through loops and conditional
branches. If the program is not too large, you can follow it easily by executing one line at a time.
There are two commands used to step through your program:
step:
Execute a single line in the program. If the current statement calls a function, the function is single stepped.
next:
Execute a single line in the program but treat function calls as a single line. This command is used to skip over function
calls.
Since C statements like printf() and scanf() are functions themselves, if you step through all your program (as opposed to
next, you'll find yourself stepping through glibc, the standard C library (which is probably not what you want!). Good debugging
makes use of next mostly. If you really want to step through a function call, it's best to set a breakpoint there and then you can
use next from inside the function.
To execute the next statement, type:
step
Each time you type a step command, gdb will then list the line that it is about to execute, with the line number on the left, so you
can see what's about to happen before it happens.
Finding out where you are and listing source code
To find out where you are at any time, type the command:
where
This will show you the current line number. For example, a line like this:
#0
foo () at foo.f:12
shows that the execution of our program is currently at a location that corresponds to line 12 in the Fortran source file, foo.f.
You can display a few lines of your source program around the current location by using the command:
list
This will list 10 lines of source roughly centred on your current line number. If you haven't started to debug yet, it will list the first 10
lines of source code. If you type list again, it'll print the next 10 lines of source code. You can also type:
list 25
and this will list 10 lines of source code centred on line 25. Typing list again will list the next 10 lines of source code. You can also
specify a range of lines to be listed. For example, to list lines 10 through 24 in the current program, you'd type:
list 10,24
If there is a function in your program named endpoints(), you can list 10 lines centred on the start of endpoints() by:
list endpoints
If you're listing lines and decide you want to see the 10 lines previous to the 10 lines you just displayed:
list Suppose you set a breakpoint:
break 55
and gdb responds with:
Breakpoint 1 at 0x8048540: file program3.c, line 55.
You can list the lines centred around that address by specifying the asterisk (for address). It will list the 10 lines centred around
the source code line containing that address.
list *0x8048540
Using The GNU GDB Debugger:
Debugging A Running Process
How To Attach GDB To An Executing Program
So far, we've debugged executables, with and without core files. However, we can debug processes too. Think about that -- we
can debug a process that has been started separately from the debugger. There are two ways of doing this: Using command line
arguments and using the attach command.
Download and read beer-process.c and its Makefile. Compile it, and run it as a background job in one console (or xterm). It'll
simply print out the number of bottles of beer on the wall:
$ ./beer-process
[1] 17399
$ 100000 bottles
99999 bottles of
99998 bottles of
99997 bottles of
&
of beer
beer on
beer on
beer on
on the wall.
the wall.
the wall.
the wall.
With Command Line Arguments
With the beer process running one console, start GDB in another console with an argument list of the executable and the process
ID. The process ID should've been printed when you started the background process:
$ gdb beer-process 17399
Attaching to program: code/running_process/beer-process, process 17399
0x410c64fb in nanosleep () from /lib/tls/libc.so.6
(gdb)
Chances are overwhelming good that the process is in GoToSleep(). Print out a backtrace and take a look at the stack:
(gdb) bt
#0 0x410c64fb
#1 0x410c6358
#2 0x0804841f
#3 0x080483e0
in
in
in
in
nanosleep () from /lib/tls/libc.so.6
sleep () from /lib/tls/libc.so.6
GoToSleep () at beer-process.c:32
main () at beer-process.c:14
Aside: Note that GoToSleep() calls the C library function sleep(), and sleep(), in turn, calls the system call
nanosleep(). As you know, all library functions (glibc on Linux) do their job by calling system calls. I'm a little surprised to see
the library and system functions listed in the call stack since I'm not using a debugging version of glibc. Weird.
At this point, the backtrace should be very familiar to you. But there's an important distinction. We didn't run this program from
within GDB. We ran it from the command line, and then had GDB attach to an already running process.
Look at the output of the beer process: you should notice that the process has stopped! Whenever GDB attaches to a running
process, the process is paused so you can get a handle on what the call stack looks like. Let's do some interesting things.
In my output above, i=9997. Yours is probably different, but nevertheless, you should be able to follow along with me. Let's verify
the value of i by selecting the stack frame for main() and looking at its value:
(gdb) frame 3
#3 0x080483eb in main () at beer-process.c:15
15
GoToSleep();
(gdb) print i
$1 = 99997
No surprises here. As you'd expect, we can use next and step (which takes us out of nanosleep() and sleep()
respectively, putting us into GoToSleep()):
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x410c6358 in sleep () from /lib/tls/libc.so.6
(gdb) step
Single stepping until exit from function sleep,
which has no line number information.
GoToSleep () at beer-process.c:34
34
}
(gdb) bt
#0 GoToSleep () at beer-process.c:34
#1 0x080483eb in main () at beer-process.c:15
Looking at the code, the next things to happen are that i will be decremented and then PrintMessage() will print 99996
bottles of beer on the wall. However, suppose we wanted more beer? Let's change to the stack frame for main()
(where i lives) and change the number of beers on the wall.
(gdb) frame 3
#3 0x080483eb in main () at beer-process.c:15
15
GoToSleep();
(gdb) set var i = 99999999
Now quit GDB. When GDB detaches from the process, the process will continue along its merry way. We could also use the
detach command to detach from the process without quiting GDB; I'll explain detach in the next session.
(gdb) quit
The program is running.
Quit anyway (and detach it)? (y or n) y
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: code/running_process/beer-process,
process 17399
but with the new value for i:
$ ./beer-process
[1] 17399
$ 100000 bottles
99999 bottles of
99998 bottles of
99997 bottles of
99999998 bottles
99999997 bottles
99999996 bottles
99999995 bottles
99999994 bottles
&
of beer
beer on
beer on
beer on
of beer
of beer
of beer
of beer
of beer
on the wall.
the wall.
the wall.
the wall.
on the wall.
on the wall.
on the wall.
on the wall.
on the wall.
I hope you're impressed by this! We attached GDB to a process that was already running. The process halted and we were able
to do everything that we would've been able to do had we started the process from within GDB. Now that's power!
One non-debugging use I've had for this in the past is with scientific programming. I had PDE solvers and Monte Carlo
applications that would run for a very long time. Whenever I wanted to take a look at how my simulation was doing or what some
of the intermediary answers looked like, I'd attach to the process using GDB and inspect my variables. This was a much better
option than simply printing everything of interest out, which could've possibly have taken hundreds of megs of disk space!
With The Attach Command
We can also debug an already running process using GDB's attach command to attach to a running process. Again, once
attached, we can use the detach command to detach from the process.
If you quit the running background process from the previous section, restart beer-process in the background. Start GDB with no
command line arguments. But use the attach command to attach to the running process.
$ gdb
(gdb) attach 17399
Attaching to process 17399
Reading symbols from code/running_process/beer-process...done.
0x410c64fb in nanosleep () from /lib/tls/libc.so.6
(gdb)
As before, the process should halt. This is when you do whatever it is you want to do with the process: debug, snoop, spy, modify,
etc. When you're done futzing around, quit GDB:
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: code/running_process/beer-process,
process 17399
As before, once you detach from the process, it'll continue running.
Processes Without Debugging Symbols
As with debugging executables and corefiles, it's only convenient to debug processes that were started from executables with
debugging information compiled into them. To see this in action, strip the executable and run it in the background again:
$ strip beer-process
$ ./beer-process &
[1] 32262
$ 100000 bottles of beer on the wall.
99999 bottles of beer on the wall.
99998 bottles of beer on the wall.
Debug the process and look at the call stack:
$ gdb
(gdb) attach 32262
Attaching to process 32262
Reading symbols from code/running_process/beer-process...(no debugging symbols found)...done.
(gdb) bt
#0 0x410c64fb in nanosleep () from /lib/tls/libc.so.6
#1 0x410c6358 in sleep () from /lib/tls/libc.so.6
#2 0x0804841f in ?? ()
#3 0x00000003 in ?? ()
#4 0x0001869d in ?? ()
#5 0xbffff7b8 in ?? ()
#6 0x080483eb in ?? ()
#7 0x0001869d in ?? ()
#8 0x0001869d in ?? ()
#9 0xbffff844 in ?? ()
#10 0x4102e7f8 in __libc_start_main () from /lib/tls/libc.so.6
#11 0x41150fcc in ?? () from /lib/tls/libc.so.6
Exercises
1. Suppose you're playing a game that you have source code for, like Doom, Nethack, or Duke Nukem 3D. How can you use
GDB to cheat, like giving yourself extra health? If you wrote your own game, can you protect the integrity of networked
games from people who would cheat like this? What kinds of things could you do?
2. Now suppose you're playing a game for which you do not have the source code. Can you still cheat in this manner? If so,
how would you go about it?
how would you go about it?
3. Do a Google search on an application called "kcheat". Read the documentation. This person, in effect, wrote a debugger. If
you have spare time, download the source and try to learn how it works. Browse the man page for the function ptrace().
4. From the previous exercise, GDB could be considered as a "front end" to ptrace() system call. Look at ps aux. Do
you see any processes that, if attached to with GDB, would be a security issue? Could cause a system to go down? Cause
filesystem corruption? You probably have a process called "init" that has a process id of 1. Try to attach to it. Now become
root and try to attach to it. There are some things that even root can't do!
Using The GNU GDB Debugger:
Debugging Ncurses Programs
Ncurses
Activities like printing characters to a screen, moving the cursor, and changing the color of character output are collectively known
as screen handling. By its nature, screen handling is very terminal dependent, however, the terminfo and termcap mechanisms
were devised to provide terminal independent screen handling. The curses library (a pun on the term "cursor optimization") was
created to provide a screen handling API for C programmers. The goal of curses was to provide a fast, portable, and terminal
independent C API to handle device dependent terminal codes.
Curses has a very long and twisted history. However, the most commonly used modern implementation of the library is called new
curses, or ncurses, for short, which is maintained by Thomas E. Dickey. Ncurses is a GNU project released under an MIT style
licence and is used under nearly all modern Unixes including GNU/Linux, and Mac OS X. There are now many extensions to
ncurses which includes panels, menus and even a full featured widget set: the Curses Development Kit (CDK).
A Sample Ncurses Debug Session
To follow along, download ncurses1.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
// ncurses1.c
#include<ncurses.h>
#include<stdlib.h>
#include<time.h>
unsigned int Seeder(void);
int Irand(int low, int high);
void Print_A_Character(void);
int main(void)
{
atexit( (void *)endwin );
initscr();
Seeder();
for (int i = 0; i < 500000; ++i)
Print_A_Character();
return 0;
}
void Print_A_Character(void)
{
int x = Irand(1, COLS);
int y = Irand(1, LINES);
unsigned ascii = Irand('A', 'z');
mvaddch(y, x, ascii);
refresh();
}
// ASCII dependent
int Irand(int low, int high)
{
return low + (int)( (double)(high-low) * rand()/(RAND_MAX + 1.0) );
}
unsigned int Seeder(void)
{
time_t seed;
time(&seed);
srand((unsigned)seed);
return seed;
}
Compile and run the program. It should fill your console (or xterm) with characters. It has a bug though: the top row and first column
seem to be devoid of characters:
Since the probability of that happening is miniscule (and gets smaller with each passing second), there must be a bug in the
program.
You need to do a bit more to use GDB with a program that uses ncurses. The problem is that GDB's I/O is intermixed with the
program's I/O. Once you get used to it, this is not normally a problem. But when the program performs screen handling, it
becomes difficult, if not impossible, to keep track of your debugging session. To see this in action, start GDB on the executable,
set a breakpoint at Print_A_Character(), and run the program.
$ gdb debugging_ncurses
(gdb) break Print_A_Character
Breakpoint 1 at 0x80486fd: file debugging_ncurses.c, line 26.
(gdb) run
Starting program: code/ncurses/debugging_ncurses
Breakpoint 1, Print_A_Character () at debugging_ncurses.c:26
26
int x = Irand(1, COLS);
Now issue continue 50 a few times. You should see a big mess. Here's what I see:
Quit GDB when you've had enough. Clearly, we need a way to separate GDB's I/O from the program's I/O when screen handling
is done.
Separating the Input/Output
You'll need two terminals (either two consoles or two xterms): One for the program's I/O and another for GDB's I/O. Separating out
the two I/O will resolve the problem nicely. I'll be using the word `xterm', but the same thing applies to all non-login terminals like
rxvt and eterm, and login terminals like virtual consoles.
1. Go to the first xterm and find its device file using either tty or who am i. This will be the xterm with GDB's I/O.:
$ tty
/dev/pts/1
$ who am i
p
pts/1
May 26 12:44 (:0.0)
2. Go to the second xterm and find its device file. This will be the xterm with our program's I/O:
$ tty
/dev/pts/4
3. Go back to the first xterm and start a debugging session. Set a breakpoint at Print_A_Character().
$ gdb debugging_ncurses
(gdb) break Print_A_Character
Breakpoint 1 at 0x80486fd: file debugging_ncurses.c, line 26.
(gdb)
4. GDB's tty command instructs GDB to redirect the program's I/O to another terminal. The argument to tty is the device
file of the terminal you wish the program I/O to go. In this case, I want the program's I/O to go to the second xterm, pts/4. If
you're following along, use whatever device file you obtained in step 2:
(gdb) tty /dev/pts/4
(gdb)
5. Lastly, go to the second xterm (that contains the program's I/O) and tell the shell to sleep for a long time. This is so that
5. Lastly, go to the second xterm (that contains the program's I/O) and tell the shell to sleep for a long time. This is so that
anything we type in that window will be sure to go to our program rather than the shell. The amount of time is arbitrary, but
pick a time that's longer than you suspect the debugging session will last. This tells the shell to "do nothing" for 100000
seconds:
$ tty
/dev/pts/4
$ sleep 100000
6. Go back to the first xterm which is running GDB and debug to your heart's content. When you're done, you can go back to
the program output window and slap it with a control-c to break out of the sleep.
Using The GNU GDB Debugger:
Other Stuff
Official GDB Sources
The official GNU GDB page is at http://www.gnu.org/software/gdb/download/. You can download versions of GDB from the
current developer's CVS all the way to the version released back in 1988!.
There are a number of mailing lists for GDB, including:
gdb-announce: a read-only low volume list for the posting of announcements about releases or important events.
gdb: a list for general discussion about GDB.
gdb-patches: patch submissions and discussion. All patch submissions and submission discussion goes here.
gdb-prs: Mailing list for discussing bugs submitted to the bug reporting database.
gdb-testers: a list for the announcement of development snapshots and the reporting of test results.
gdb-cvs: is where CVS commit messages go when things are checked into the GDB CVS repository.
src-cvs: is where CVS commit messages for the top-level files and shared directories go.
gdbadmin: A read-only list for cron log messages and other dull boring stuff.
gnu.gdb.bug: (mail relay bug-gdb) is the public GDB news group.
GNU's official GDB user manual in html and pdf.
The GDB Internals Manual.
Formats For This Document
This document is available in the following formats:
Multi-page HTML: The canonical, and most up to date version (see below).
PDF: This should open in most browsers. If you have a slow connection, you may want to try the bzipped version below.
bzipped PDF: To download the file with your browser, right-click, and select "Save link as...". To extract the file gdb.pdf.bz2,
on Unix/Linux/OSX systems simply do (from the command line): bunzip2 gdb.pdf.bz2 In Windows, you can extract the file
with the 7zip utility.
As noted above, the multi-page HTML version is apt to be the most up to date document. Due to time constraints, I will not
necessarily always have the PDF updated at the same time I update the HTML-doc (though I will do my best to keep the two in
sync). If in doubt, view the changelog, where I will indicate in each update if I have updated the PDF.
Note:The PDF was generated from the multi-page HTML version using the wkhtmltopdf utility. While the program
generally does a decent job of the conversion, there are some slight issues, such as varying text-size from section to
section. If you find any issues with the text of the PDF, please let me know.
Other GDB Tutorials
An excellent tutorial by Norm Matloff. Highly recommended.
A very good question and answer style tutorial, called "RMS's GDB Tutorial". I think this guy (Ryan Michael Schmidt)
probably gets mistaken for the other RMS quite often.
A tutorial that focuses on C++.
Kudos
I've received some great email from around the world. I have a keen interest in other peoples' cultures, their likes, dislikes, what
they do, who they are. I'd love to fly to each country and make each one of these people my personal friend. But I can't, so I'll
simply post some of the fantastic "kudos" that this page has generated.
If you want your email taken down (or don't want it posted) let me know and I'll be happy to oblige. I'm just grateful to get any email
kudos at all. Feel free to send me your homepage, pic, or blog and I'll post them here if you like.
USA - Abhishek Sharma: homepage - 2006 Jun 21 06
India - Sri Charan: blog, blog entry - 2006 Apr 20
India - Vihan Pandey - 2006 May 01
USA (Santa Clara) - Rayees Shamsuddin - 2005 Aug 29
Brazil - Hilton Fernan - 2005 Aug 02
Brazil - Fábio Luiz - 2005 Jul 22
India - Ram - 2005 Mar 04
Canada - Mark Lord - 2005 Jan 21