Performance profiling — CPU, memory, I - O profiling: Question bank

Definition Mini-glossary — the words every item below leans on

Read these once before the questions; they are used everywhere and never re-explained inside a reveal.

Wall time ( $T_{wall}$ ): the real, on-the-clock elapsed time from start to finish — what a stopwatch on the wall would read, waiting included.
CPU time ( $T_{CPU}$ ): time the processor actually spent executing your program, split into user (your code) and sys (kernel work on your behalf). It excludes any waiting.
Wait time ( $T_{wait}$ ): the leftover, $T_{wait} = T_{wall} - (T_{CPU(user)} + T_{CPU(sys)})$ — the part of the clock spent idle, blocked on disk, network, a lock, or sleep.
Self (exclusive) time: time spent inside a function itself, excluding the functions it calls. This is where cycles actually burn.
Cumulative (inclusive) time: a function's self time plus all of its descendants — so a thin wrapper can show a huge cumulative number while doing almost nothing itself.
Cumulative width (in a flame graph): the horizontal length of a bar; width $=$ cumulative time. The widest bars are the hotspots; a bar's height is only call-stack depth, not time.
In the sampling-error formula $S E = p (1 - p) / N$ that appears below: $p$ is the true fraction of CPU time spent in a function (the thing we are estimating), and $N$ is the number of samples the profiler took. Its derivation lives on the parent note.

True or false — justify

The word "true" or "false" alone earns nothing here — the reasoning is the answer.

A deterministic profiler and a sampling profiler will always rank the hotspots in the same order.

False. A deterministic profiler hooks every call, so it inflates call-heavy functions (thousands of tiny calls) disproportionately, distorting the ranking; a sampling profiler weights by wall-clock presence and often ranks such functions lower.

If a CPU profiler reports your code uses very little CPU, the program is fast.

False. Low CPU can mean the program is I/O-bound — waiting on disk or network — so wall-clock time is huge while the CPU sits idle. The CPU profiler simply cannot see waiting.

A function with the highest cumulative (inclusive) time is the best place to optimize.

False. Cumulative time includes all descendants, so a thin wrapper that just calls the real worker shows a huge cumulative number while doing almost no work itself. Sort by self time to find where cycles actually burn.

Sampling profilers give wrong answers because they only look occasionally.

False (for the hotspots that matter). Sampling is a Monte-Carlo poll: a function present in

X %

of snapshots uses

\approx X %

of CPU, and the standard error

S E = p (1 - p) / N

(with

p

the true CPU fraction,

N

the sample count) shrinks fast, so big hotspots resolve in under a second.

In a flame graph, a taller stack means a slower function.

False. Height is call-stack depth, not time — it shows who called whom. Cumulative width is time. A short, very wide bar is a much bigger hotspot than a tall, narrow tower. See Flame Graphs.

Amdahl's Law says any optimization eventually makes the program infinitely fast if you try hard enough.

False. As speedup factor

s \to \infty

, overall speedup approaches only

S = \frac{1}{1 - p}

, capped by the untouched fraction

1 - p

. Optimizing a part that is 20% of runtime maxes out at

1.25 \times

no matter what. See Amdahl's Law.

Big-O tells you which line to optimize after profiling.

False. Big-O predicts scaling with input size, not where wall-clock time actually goes on your real workload — a

O (n^{2})

loop over 5 items is irrelevant next to an

O (n)

database call. Profile the real run; use Big-O Complexity to reason about how the measured hotspot grows.

A microbenchmark that loops a function a million times gives its real-world speed.

False. Loops warm caches, trigger JIT warmup, let the compiler dead-code-eliminate unused results, and strip out the real I/O — so the number is optimistic and unrepresentative. This is benchmarking, not profiling the real path.

Turning on a profiler never changes the numbers it reports.

False. A tracing profiler adds 2–10× overhead concentrated on call boundaries, so heavily-called functions look relatively slower than they are. Use a low-overhead sampling profiler when overhead would distort the ranking.

Spot the error

Each line below contains a flawed reasoning move. Name the flaw and the fix.

"The nested triple loop looks terrifying, so I rewrote it first."

The error is optimizing by appearance, not measurement — the loop may run once on tiny data. Fix: profile, then optimize by measured self (exclusive) time.

"db_query has 75% cumulative width and CPU is low, so I'll rewrite it in C to speed up the CPU work."

The error is treating an I/O wait as CPU work. Flat, wide (75% cumulative time), low-CPU = waiting on the database; rewriting the client language does nothing. Fix the query (index it, batch it) — see N+1 Query Problem.

"Wall time is 100 ms and CPU time is 95 ms, so it must be I/O-bound."

The error is the direction of the inequality.

T_{wall} \approx T_{CPU}

(wait time near zero) means CPU-bound; you'd need

T_{wall} ≫ T_{CPU}

(waiting dominating) to call it I/O-bound.

"I profiled the dev machine with 10 rows in the table, and it was instant, so production is fine."

The error is a non-representative workload. The N+1 query cost and I/O scale with real data volume; 10 rows hides the very bottleneck production exposes. Profile a realistic end-to-end load.

"Memory usage climbs forever, but the garbage collector runs, so there's no leak."

The error is assuming GC frees everything. In a managed language a leak is memory that is still reachable but forgotten — a growing global cache or dangling listener — which GC will never collect. See Memory Management & Garbage Collection.

"Adding a cache made this endpoint faster, so I'll cache everything."

The error is ignoring the measured slice. If the cached call was only 5% of runtime, Amdahl caps the win near

1.05 \times

while you add cache-invalidation bugs. Cache the measured big slice — see Caching Strategies.

Why questions

Why do we profile before optimizing instead of just fixing the obvious slow part?

Because developer intuition about bottlenecks is famously wrong, and Amdahl's Law means effort spent on a small fraction

p

yields at most

\frac{1}{1 - p}

overall — you must find the big $p$ first or you gain nothing.

Why does a sampling profiler need a different tool for I/O than for CPU?

A sampling profiler asks "what CPU instruction is running now?"; during I/O the CPU runs nothing of yours, so those waits are invisible. You need syscall tracing (strace), iostat, or distributed traces to see the wait — see Observability — Logs, Metrics, Traces.

Why is "self time" the right sort key for finding where to optimize?

Self (exclusive) time is the work done inside a function excluding its callees, so it points at the exact frame burning cycles rather than at wrappers whose big cumulative number is really their children's work.

Why does sampling error shrink as

N

rather than as

N

Because each of the

N

samples is an independent Binomial trial, and the standard error of the estimated CPU fraction is

p (1 - p) / N

— variance falls linearly with

N

, so the error (its square root) falls as

N

, giving fast but diminishing returns.

Why can time ./prog be called "the cheapest profiler you own"?

Because its real, user, sys outputs decompose

T_{wall} = T_{CPU(user)} + T_{CPU(sys)} + T_{wait}

(where

T_{wait} = T_{wall} - (user + sys)

) for free, instantly telling you whether to reach for a CPU profiler or an I/O tool — the single most important distinction.

Why does the N+1 query problem show up as an I/O bottleneck rather than a CPU one?

Each of the N extra queries is a network round-trip to the database; the program spends its time waiting for responses, so wall-clock balloons while CPU stays near idle — the classic I/O signature.

Edge cases

If the optimizable part is

p = 0

(touches nothing on the hot path), what overall speedup does Amdahl predict?

S = \frac{1}{( 1 - 0 ) + 0/ s} = 1

. Zero — no speedup at all, because you optimized code that never runs on the measured path.

If a function is

100%

of runtime (

p = 1

) and you speed it

s

times, what is the overall speedup?

S = \frac{1}{0 + 1/ s} = s

. The whole program speeds up by exactly

s

, the only case where local and global speedup coincide.

What does a sampling profiler report for a program that spends all its time sleep()-ing?

Essentially no CPU samples land in your code — it appears to do "nothing" — yet wall-clock time is huge. This is the pure degenerate I/O/wait case a CPU profiler cannot represent.

You take only

N = 1

sample of a hotspot with true CPU fraction

p = 0.25

. Is the estimate meaningful?

No. With one sample the estimate is either 0 or 1 and

S E = 0.25 \cdot 0.75/1 \approx 0.43

— as large as the value itself. Sampling only becomes trustworthy once

N

is large enough that

S E

is a small fraction of

p

What happens to a deterministic profiler's reported ranking when a function makes millions of trivial calls versus one function doing heavy math in a tight loop?

The call-heavy function is inflated by per-call hook overhead and may outrank the genuinely heavy one — a distortion that vanishes under sampling, which is why you switch tools for call-heavy hot code.

If wall time equals CPU time and both are tiny, but users report slowness, where is the time going?

Not in this process at all — it is upstream or downstream (a slow dependency, queue wait, or network hop). You need end-to-end tracing across services, not a single-process profiler, to locate it — see Observability — Logs, Metrics, Traces.

For a memory leak in a non-GC language like C, what distinguishes it from a GC-language leak?

A C leak is memory that became unreachable but was never freed (a lost pointer); a GC-language leak is memory that stays reachable through a forgotten reference. Same symptom (memory grows), opposite root cause — and only the GC one is invisible to a leak detector that hunts unreachable blocks.

Recall One-line self-test before you leave

Cover every answer above; if you can justify each in your own words in one sentence, you have internalized the traps. The single sentence that ties them together: measure the big slice with the right stopwatch, then read self (exclusive) time, not appearances.