GDB debugging — breakpoints, watchpoints, step, next, backtrace: Worked examples

See the parent: GDB debugging (topic note).

The scenario matrix

Before solving anything, let us list every kind of question this topic can throw. Think of it as a grid: each row is a class of situation, and every worked example below fills in one row.

#	Case class	The knob that changes	Degenerate / limiting edge
C1	Software watchpoint cost	$N$ , $c$ , $T_{0}$ all finite	$c \to 0$ (hardware) ⇒ $S \to 1$
C2	Hardware watchpoint limit	number of watched bytes	5th watchpoint ⇒ forced software
C3	`step` vs `next` trace	call present vs absent	line has no call ⇒ they behave identically
C4	Breakpoint mechanics	byte substitution at address $A$	instruction pointer must back up by 1
C5	Crash triage with `bt`	which frame holds the bug	bug is in the caller, not the crash line
C6	Optimization erases a variable	`-O0` vs `-O2`/`-O3`	`<optimized out>` at high `-O`
C7	Real-world word problem	pick hardware vs software live	choose the cheaper tool under a time budget
C8	Exam twist	breakpoint "didn't fire"	pending symbol / inlining / zero hits

Every cell below is covered. Signs and limits live inside the cost-model rows (C1, C2, C7); the degenerate control-flow case is C3's edge; the zero-hit degenerate case is C8.

C1 — Software watchpoint cost (the core formula)

Worked example

[Case C1] Predict the slowdown A program runs in $T_{0} = 2 s$ executing $N = 5 \times 1 0^{7}$ source lines. You put a software watchpoint on an out-of-scope variable; each line now pays a trap+re-evaluate cost $c = 2 μ s = 2 \times 1 0^{- 6} s$ . How long does it take, and what is the slowdown $S$ ?

Forecast: Guess before computing — will it be roughly 2×, 50×, or 500× slower? (Trap cost per line vs normal cost per line decides it.)

Extra time from traps $= N \cdot c = (5 \times 1 0^{7}) (2 \times 1 0^{- 6}) = 100 s$ . Why this step? The model says the only new work is one trap per line; multiply the two.
Total time $T_{wp} = T_{0} + N c = 2 + 100 = 102 s$ . Why this step? Normal execution time $T_{0}$ is unavoidable; add the trap overhead on top.
Slowdown $S = T_{wp} / T_{0} = 102/2 = 51$ . Why this step? Slowdown is the ratio of new time to old time — the dimensionless "how many times worse".

Verify: Use the closed form $S = 1 + N c / T_{0} = 1 + 100/2 = 1 + 50 = 51$ . ✓ Matches. Units: $N c$ is (lines)(s/line) = s, divided by s ⇒ dimensionless. Our forecast of "tens of times" was right.

C2 — Hardware watchpoint limit → forced software fallback

Worked example

[Case C2] The 5th watchpoint You already watch four 4-byte ints (all in hardware). The same loop from C1 runs in $T_{0} = 2 s$ , $N = 5 \times 1 0^{7}$ , $c = 2 \times 1 0^{- 6} s$ . You add a fifth watchpoint. What happens to the run time?

Forecast: Will the fifth one be free (like the first four) or catastrophic?

Count usable debug registers $= 4$ (DR0–DR3 full; DR6/DR7 are status/control, not slots). A fifth has no register. Why this step? The hardware limit is a count of address slots, not a size — four watchpoints, done.
Fifth falls back to software ⇒ whole program single-steps, cost model becomes C1. Why this step? One software watchpoint is enough to force single-stepping of every line.
Run time $= T_{0} + N c = 2 + 100 = 102 s$ — same slowdown $S = 51$ as C1. Why this step? The four hardware watches stay ~free; the single software watch dominates.

Verify: $S = 1 + N c / T_{0} = 1 + 50 = 51$ . ✓ The catastrophic jump comes from crossing the count limit — the fifth watch is not "a little slower", it is 51× slower. Fix: free one register (delete <n>) so all five never coexist in hardware, or watch fewer bytes.

C3 — `step` vs `next`: trace them, including the degenerate edge

Intuition What the figure below shows (read this first)

Figure s01 draws main's three source lines as a tall column on the left and square's two inner lines as a shorter column on the right. Two arrows race downward. The black arrow (next) runs straight down the left column — line 2 → line 3 → line 4 — and never touches the right column: it executes square invisibly and stays at main's level, so it makes only 2 stops. The red arrow (step) veers off the left column at line 2 into square's two right-column lines, then returns left — so it makes 4 stops. The single takeaway to carry away: the two arrows diverge only at line 2, the one line that contains a call; on the call-free line 3 both arrows take the identical one-hop step down, which is exactly why step and next are indistinguishable there.

Figure — GDB debugging — breakpoints, watchpoints, step, next, backtrace

Worked example

[Case C3] Count the stops

1  int main() {
2      int a = square(3);   // square() has 2 lines inside
3      int b = a + 1;       // NO call on this line
4      return b;
5  }

You break at line 2. (a) How many times must you press next to reach line 4? (b) How many steps to reach line 4? (c) On line 3 (no call), do step and next differ?

Forecast: Will step need more presses than next here? Why?

next from line 2: runs square() fully, lands on line 3. One more next: line 4. Total 2. Why this step? next = No-dive; each press advances exactly one line at this level.
step from line 2: dives into square — press 1 enters square line, press 2 next inner line, press 3 back to line 3, press 4 to line 4. Total 4. Why this step? step descends, so it must walk the two inner lines of square before returning.
Line 3 has no call ⇒ step and next both just go to line 4. Identical. Why this step? The dive/skip distinction only exists when there is something to dive into — the degenerate case where the two commands collapse into one.

Verify: next presses $= 2$ , step presses $= 4$ , difference on line 3 $= 0$ . The extra $4 - 2 = 2$ step presses equal exactly the 2 inner lines of square. ✓ See Stack frames & calling conventions for why "descend" means "push a new frame".

C4 — Breakpoint mechanics: the byte and the pointer

Worked example

[Case C4] Where is the instruction pointer? A breakpoint is set at address $A = 0x401136$ . The INT3 byte is one byte wide. After the trap fires, before GDB fixes anything, what value is in the instruction pointer, and what must GDB do?

Forecast: Does the pointer sit at 0x401136 or one past it?

0xCC is 1 byte; the CPU increments the pointer as it fetches. After executing it, pointer $= A + 1 = 0x401136 + 1 = 0x401137$ . Why this step? The fetch-and-increment happens before the trap is delivered.
GDB restores the saved original byte at $A$ , then subtracts 1 from the pointer: $0x401137 - 1 = 0x401136$ . Why this step? So that when you continue, the CPU re-executes the real instruction, not the middle of it.

Verify: $A + 1 - 1 = A = 0x401136$ . ✓ In decimal: $0x401136 = 4198710$ , and $4198710 + 1 - 1 = 4198710$ . The net offset applied by GDB is exactly $- 1$ .

C5 — Crash triage: the bug lives one frame up

Worked example

[Case C5] Who handed over the NULL?

Program received signal SIGSEGV, Segmentation fault.
0x0000000000401180 in process (p=0x0) at list.c:88
(gdb) bt
#0  process (p=0x0)  at list.c:88
#1  main ()          at list.c:120

The crash line dereferences p, which is 0x0 (NULL). Which frame do you inspect to find the cause, and what command switches you there?

Forecast: Is the bug on line 88 (where it crashed) or line 120 (who called it)?

Line 88 is the symptom: dereferencing NULL raises SIGSEGV. But process was handed the NULL. Why this step? A NULL-deref is rarely a bug in the dereferencing code — it is a bug in whoever produced the NULL.
bt shows the caller is main at frame #1, line 120. Switch with frame 1. Why this step? frame <n> moves your inspection context up the stack (Stack frames & calling conventions).
In frame 1, print head reveals head was never initialized before the call. Why this step? The uninitialized head became the NULL argument — the true bug, one frame up.

Verify: Crash frame index $= 0$ , cause frame index $= 1$ , difference $= 1$ frame up. ✓ Rule of thumb: on a crash, bt first; the real bug is typically the caller, not the callee.

C6 — Optimization erases a variable

Worked example

[Case C6] <optimized out> at high -O You debug the parent's factorial. Under -O0 you print result and see numbers; under -O2 GDB prints <optimized out>. For factorial(5), what value should result hold at return, and why does -O2 hide it?

Forecast: Does the variable stop existing, or just stop being visible?

Compute the true value: $5! = 1 \cdot 2 \cdot 3 \cdot 4 \cdot 5 = 120$ . That is what result returns. Why this step? We need a concrete expected value to know the debugger would show 120 under -O0.
Why -O2 hides it: the optimizer keeps result in a register transiently or folds the loop; there is no stable memory slot named result for GDB to read. Why this step? <optimized out> means "the value exists conceptually but has no addressable home right now" — see Optimization levels -O0 -O2 -O3.
Fix: rebuild gcc -g -O0 bug.c so debug symbols map result to a real location. Why this step? -O0 forbids the deletions/register-only tricks that erase the variable's home.

Verify: $5! = 120$ . ✓ The value is correct in both builds; only its visibility differs. Debugging is about visibility, so -O0 wins.

C7 — Real-world word problem: choose the tool under a time budget

Worked example

[Case C7] Which watchpoint fits the deadline? A test harness normally finishes in $T_{0} = 0.5 s$ , executing $N = 2 \times 1 0^{8}$ lines. Trap cost $c = 5 \times 1 0^{- 7} s$ . Your CI job times out at 10 s. You must watch one 4-byte counter. May you use a software watchpoint, or must it be hardware?

Forecast: Software adds $N c$ seconds — will that exceed the 10 s budget?

Software overhead $N c = (2 \times 1 0^{8}) (5 \times 1 0^{- 7}) = 100 s$ . Why this step? This is the pure trap cost the software path forces onto you.
Software total & slowdown $T_{sw} = T_{0} + N c = 0.5 + 100 = 100.5 s$ , so $S_{sw} = 1 + N c / T_{0} = 1 + 100/0.5 = 201$ . Why this step? Compare against the budget: $100.5 s > 10 s$ ⇒ times out.
Hardware total & slowdown the counter (4 bytes ≤ 8) fits one free debug register, so $N c \to 0$ : $T_{hw} = T_{0} + 0 = 0.5 s$ , and $S_{hw} = T_{hw} / T_{0} = 0.5/0.5 = 1$ . Why this step? Making the dimensionless slowdown explicit ( $S_{hw} = 1$ ) shows hardware costs nothing extra — exactly the $c \to 0 \Rightarrow S \to 1$ limit from the matrix.

Verify: Software $S_{sw} = 201 \Rightarrow 201 \times 0.5 = 100.5 s$ ; hardware $S_{hw} = 1 \Rightarrow 1 \times 0.5 = 0.5 s$ . Since $100.5 > 10 \geq 0.5$ , only hardware fits. ✓ GDB picks hardware automatically when a register is free — verify with info watchpoints showing hw.

C8 — Exam twist: the breakpoint that "never fired" (two distinct symptoms)

Worked example

[Case C8a] Symptom A — <PENDING> address, 0 hits (unresolved symbol)

Num  Type        Disp Enb Address   What
1    breakpoint  keep y   <PENDING> in helper
     breakpoint already hit 0 times

The Address column says <PENDING>. Name the causes and give the diagnosis + fix for each.

Forecast: Is GDB broken, or was helper's address never resolvable?

Read the Address column: it says <PENDING>, i.e. GDB never bound the breakpoint to a real address. This is a symbol problem, not a code-path problem. Why this step? <PENDING> is the definitive fingerprint of an unresolved symbol — it rules out the reachability case in one glance.
Cause A1 — missing/late symbols: the binary was built without -g, or helper lives in a shared library not yet loaded so its address is unknown until the object maps in. Why this step? GDB can only resolve helper once a symbol table containing it is present in memory.
Cause A2 — inlining: at -O2/-O3 the compiler pasted helper's body into its callers, so there is no standalone helper entry address to trap (Optimization levels -O0 -O2 -O3). Why this step? An inlined function has no address of its own; the breakpoint has nothing to attach to.
Fix per cause: for A1 rebuild with -g (debug symbols) or, for a shared-library target, set breakpoint pending on so GDB binds it when the library loads; for A2 rebuild -O0 so a real helper address exists. Why this step? Each fix restores the symbol → address mapping that <PENDING> reported missing.

Verify: Observed hit-count $= 0$ with <PENDING>. After the matching fix, re-run info breakpoints: the Address column shows a real hex value and, once a run calls helper, hit-count $= 1 > 0$ . ✓ "GDB is broken" is disproved — <PENDING> was an honest report of an unresolved symbol.

Worked example

[Case C8b] Symptom B — resolved hex address, 0 hits (code path never taken)

Num  Type        Disp Enb Address            What
1    breakpoint  keep y   0x0000000000401210 in helper at util.c:12
     breakpoint already hit 0 times

The Address column shows a real hex value 0x401210, yet hit-count is still 0. The symbol clearly resolved — so why no stop?

Forecast: With a resolved address, must the breakpoint fire when you run?

Read the Address column: real hex ⇒ the symbol resolved fine. This is a reachability problem, not a symbol problem — the opposite diagnosis from C8a. Why this step? The resolved address rules out A1/A2; the fingerprint points at control flow instead.
Cause B1 — you forgot run/continue: the breakpoint is set but the program (or that stretch of it) never actually executed. Why this step? A breakpoint can only be hit while the inferior is running; a set-but-not-run breakpoint trivially has 0 hits.
Cause B2 — the branch calling helper was never taken: e.g. helper sits inside an if whose condition was false for this input, so line 12 is never reached. Why this step? A live breakpoint on unexecuted code stays at 0 hits — nothing wrong with the breakpoint, the code simply detoured around it.
Diagnose & fix: confirm you issued run; then set a breakpoint at the guarding condition (e.g. break util.c:8 on the if) and print the condition's variables to see whether the branch is entered. Feed an input that takes the branch. Why this step? You verify reachability directly instead of blaming the breakpoint — this is the scientific method: hypothesise, place a probe upstream, observe.

Verify: Observed hit-count $= 0$ with a resolved address. After feeding an input that enters the branch (and issuing run), hit-count $= 1 > 0$ . ✓ The two symptoms are genuinely distinct: <PENDING> (C8a) needs a symbol fix (-g/-O0/pending-on); a resolved-address 0-hit (C8b) needs a reachability fix (run it / take the branch).

Recall Self-test: match each answer to its matrix cell

Software watchpoint with $N = 5 \times 1 0^{7}, c = 2 μ s, T_{0} = 2$ — slowdown? ::: $S = 51$ (C1) 5th watchpoint on x86 with DR0–DR3 full — what happens? ::: Falls back to software, $S = 51$ again (C2) next presses vs step presses to cross a 2-line call — difference? ::: 2 extra step presses; on a call-free line, difference is 0 (C3) Net offset GDB applies to the instruction pointer after an INT3 trap? ::: $- 1$ (C4) On a NULL-deref crash, how many frames up is the usual bug? ::: 1 (the caller) (C5) Return value of factorial(5) regardless of -O level? ::: $120$ (C6) Software vs hardware slowdown for $N = 2 \times 1 0^{8}, c = 5 \times 1 0^{- 7}, T_{0} = 0.5$ ? ::: $S_{sw} = 201$ (100.5 s, over budget) vs $S_{hw} = 1$ (0.5 s) (C7) A breakpoint shows <PENDING> and 0 hits — what class of cause? ::: Unresolved symbol: missing -g / unloaded shared lib / inlining (C8a) A breakpoint shows a real hex address but 0 hits — what class of cause? ::: Reachability: forgot run, or the branch was never taken (C8b)

Related method: The scientific method in debugging · Deeper tooling: Memory debugging — Valgrind & AddressSanitizer.