5.3.16 · D5Build Systems & Toolchain
Question bank — Profiling — gprof, perf, Valgrind - Callgrind
Before we begin, a small vocabulary we lean on everywhere below, each defined in plain language and anchored to a picture.
The call-tree picture below makes this concrete: main inherits everyone's time as total but has almost no self time of its own.
The timeline below contrasts the two: instrumentation pays a little tax on every event (grey ticks), while sampling only peeks at the PC on each timer interrupt (arrows).
The bar picture shows why: shrink the coral (hot) block to nothing and the mint (untouched) block alone still sets the floor of the new runtime.
Finally, the reason a "small" cache miss rate can dominate — the cost per miss is enormous compared to a hit:
True or false — justify
True or false: main showing the highest time in a profile means main is the bottleneck.
False — that is its total time, which includes every callee.
main itself usually does almost nothing; optimize by self time, use total only to locate expensive subtrees.True or false: a lower-overhead profiler always gives you the more trustworthy picture.
False — low overhead (perf) means less timing distortion, but sampling is statistical and can miss rare-but-costly events; a heavy tool (Callgrind) gives exact counts. "Trustworthy" depends on the question you asked.
True or false: Callgrind reports the real number of wall-clock seconds your program took.
False — Callgrind runs a simulated CPU and reports instruction/cache events, not seconds. It ignores out-of-order execution, prefetch, and true DRAM latency; use perf for real time.
True or false: if a function is called a million times, making the function body faster is the only lever you have.
False — you can also reduce the number of calls (memoize, hoist out of the loop, batch/vectorize). High call count is itself a signal that the call frequency, not just per-call cost, may be the target.
True or false: profiling a -O0 build is fine as long as you look at relative percentages.
False — the optimizer rewrites the program: inlining deletes whole functions, vectorization reshapes hot loops. The
-O0 percentages describe code that does not exist in the release binary.True or false: a 1% cache miss rate is small enough to ignore.
False — a miss to DRAM costs ~100+ cycles versus ~4 for an L1 hit, so 1% of accesses can dominate total time. Miss rate must be weighed against miss cost, never read alone.
True or false: instrumentation profilers give exact call counts, so their reported times are exact too.
False — the call counts are exact (a counter ticks each call), but the time comes from PC (program-counter) sampling, which is statistical, and the instrumentation overhead itself distorts the timing it measures.
True or false: high IPC always means your program is fast.
False — IPC (instructions per cycle) only says the CPU isn't stalling. You can retire instructions efficiently while running a wasteful algorithm; high IPC + high instruction count means you need a better algorithm, not better memory behaviour.
True or false: two runs of the same program under perf will give byte-identical numbers.
False — perf uses timer/event interrupts and real hardware, so numbers wobble run to run. For deterministic, reproducible counts (e.g. CI regression gates) you want Callgrind.
Spot the error
"IPC is 0.4, so the algorithm is too slow — let me pick a better algorithm."
Wrong diagnosis. Low IPC means the CPU is stalling (cache misses, branch mispredicts), so the fix is better memory/branch behaviour (locality, layout). A better algorithm helps when IPC is already high but instruction count is large.
"perf report shows strlen as hot, so strlen is badly written — replace it."
The error is blaming the callee.
strlen is hot because your code calls it too often, likely in a loop where the length doesn't change. Fix the caller: hoist the length out, don't rewrite libc."I'll optimize this deeply nested loop; nested loops are obviously the slow part."
Guessing, not measuring. The loop might run only 10 times while a hidden
log() elsewhere dominates. Profile first; intuition about hot paths is usually wrong."Callgrind said function A costs 3× function B, so on my CPU A takes 3× the seconds of B."
Simulated instruction cost ≠ real seconds. Prefetching and out-of-order execution can hide much of A's cost on real silicon. Callgrind is for relative counts; confirm real timing with perf.
"gprof shows main with 95% total time — I'll rewrite main."
Confusing total with self time.
main's 95% is inherited from its callees; its own body does nearly nothing. Follow the total-time subtree down to the function with high self time."perf's overhead is ~1–5%, so I can trust it even for a function that runs for 2 microseconds total."
A function that never survives to a sampling interrupt gets zero or wildly noisy samples. Low overhead doesn't fix low sample counts; for tiny/rare code, sampling is statistically blind.
"I added -pg to profile, kept -O0 for clarity, and got a clean report."
Two errors compounded:
-pg measures the debug binary the optimizer would transform anyway, and -O0 disables inlining/vectorization. The report describes non-existent release behaviour.Why questions
Why does profiling insist "measure before you optimize" instead of just reading the code?
Because real cost is dominated by things invisible in source: cache misses, branch mispredicts, syscalls, and call frequency. The line that looks expensive rarely is the one that is.
Why does higher count accuracy tend to force higher overhead?
To count every event exactly you must instrument (insert work on each event), and that inserted work itself consumes time — so measuring more precisely disturbs the very timing you measure.
Why does perf capture more samples where the program does more work (event-based sampling)?
It counts a hardware event (e.g. cycles) and interrupts after every events, where is the sampling period — a fixed event budget you (or perf) set. Busy code burns through those events faster, triggering interrupts more often, so hot regions naturally accumulate denser samples.
Why is Callgrind valued for CI regression tests despite being 20–100× slower?
Because it is deterministic: identical input yields identical counts with no timer noise, so a genuine +2% instruction regression is distinguishable from run-to-run jitter — which perf's noise would hide.
Why do we distinguish self time from total time at all?
Self time tells you which single function to rewrite; total time tells you which subtree of calls is expensive. Optimizing by total alone sends you to
main, which you can't speed up directly.Why does adding -fno-omit-frame-pointer help perf's call graphs?
Optimizers may reuse the frame-pointer register, breaking the chain perf follows to reconstruct callers. Keeping the frame pointer preserves that chain, giving accurate caller→callee attribution.
Why can a 12% cache-miss rate explain a low IPC, making an algorithm change the wrong fix?
Each miss stalls the pipeline for ~100 cycles, so the CPU spends cycles waiting, not retiring instructions — that is low IPC. The cure is data locality (blocking, struct-of-arrays), not a new algorithm.
Edge cases
What does self time look like for a function that only calls other functions and does nothing else?
Its self time is ~zero while its total time can be large. It is a pure delegator; profiling by self time correctly skips it.
What happens to gprof's picture when a hot function gets inlined by -O2?
Its body is merged into the caller, so it may vanish from the profile — its cost is now attributed to whoever inlined it. This is exactly why
-O0 and -O2 profiles disagree.What does perf report for a function so short-lived it runs entirely between two timer interrupts?
Possibly no samples at all, making it look free. Sampling has a resolution floor; genuinely tiny/rare code needs exact tools (Callgrind) or microbenchmarks to see.
Amdahl's ceiling: if the hot function is fraction of runtime, what's the best possible total speedup even with infinite speedup of it?
At most — the un-optimized 28% caps everything. Predict this before investing effort. See Amdahl's Law & Scalability.
What if cache-references is essentially zero — is a reported miss rate meaningful?
No — miss rate is misses ÷ references, so dividing by a near-zero denominator makes the percentage unstable and meaningless. Report absolute miss counts in that degenerate case.
What does an IPC near the CPU's peak (say ~3.5 of 4) combined with a huge instruction count tell you?
The hardware is running near-optimally; the machine isn't the problem — you are simply doing too much work. The lever is a lower-complexity algorithm, not memory or branch tuning.
Related prerequisites worth revisiting for these traps: Cache Hierarchy & Locality of Reference, Branch Prediction & Pipelining, Compiler Optimization Flags (-O2, -O3, inlining), Benchmarking & Microbenchmark Pitfalls, and Debugging with gdb.
Recall One-line summary of every trap here
Measure the right thing (self vs total), with the right tool (sampling vs exact), on the right binary (-O2 -g), and read metrics together (miss rate needs miss cost; IPC needs instruction count) — never in isolation.