Performance profiling — CPU, memory, I - O profiling: Exercises

Definition Self (exclusive) time vs cumulative (inclusive) time (needed from L3 onward)

When a function $F$ runs, the profiler splits the time two ways:

Self (exclusive) time of $F$ = time spent executing the lines of $F$ itself, excluding any time spent inside functions that $F$ calls. This is where cycles actually burn inside $F$ .
Cumulative (inclusive) time of $F$ = self time of $F$ plus the entire time spent in every function $F$ calls (its descendants). So $cumulative = self + \sum (descendants)$ .

Picture it (figure below): stack the call frames. Cumulative time is the whole bar under $F$ — $F$ and everything sitting on top of it. Self time is only the sliver of that bar not covered by any child. A pure wrapper has tiny self time (thin sliver) but huge cumulative time (wide bar), because all the work is in the children stacked above it.

Level 1 — Recognition

Goal: name the concept. No arithmetic.

L1.1 A profiler slows your program by 5× because it hooks every function call and return, counting each exactly. What kind of CPU profiler is this — deterministic or sampling?

L1.2 time ./prog prints real 8.0s, user 0.4s, sys 0.3s. Total CPU is $0.4 + 0.3 = 0.7$ s, but wall-clock (real) is $8.0$ s. Is this program CPU-bound or I/O-bound?

L1.3 In a flame graph, what does the width of a bar represent, and therefore which bar is the hotspot?

Recall Solution — L1

L1.1 Deterministic (tracing) profiler. The giveaways: (a) it hooks every call/return, and (b) it slows the program by a large factor (2–10×). A sampling profiler would instead peek at intervals and add almost no overhead. Note two properties of a sampling profiler we will lean on in L5: it has low overhead (peeks at intervals, does not hook every call), and it typically attaches to an already-running process without instrumenting or recompiling the code.

L1.2 I/O-bound. Using the decomposition from the definition above, $T_{wait} = T_{wall} - (T_{CPU(user)} + T_{CPU(sys)})$ . Here $T_{wall} = 8.0$ s but total CPU is only $0.7$ s, so $T_{wait} = 8.0 - 0.7 = 7.3$ s of pure waiting (disk/network/lock/sleep). A CPU profiler would show your code "idle" and mislead you.

L1.3 Width = time (cumulative time of that call frame). The widest bar is the hotspot — the frame the program spends the most time inside. See the flame-graph figure in L3.1 below and Flame Graphs.

Level 2 — Application

Goal: plug numbers into Amdahl's Law and the sampling error formula.

L2.1 A request takes $200$ ms. Of this, $120$ ms is database access, which you make $3 \times$ faster. What is the overall speedup $S$ , and the new request time?

L2.2 A function is $p = 0.40$ of runtime. You optimize it infinitely ( $s \to \infty$ ). What is the maximum possible overall speedup?

L2.3 You want to resolve a hotspot whose true fraction is $p = 0.20$ to a standard error of $S E = 0.01$ (one percentage point). How many samples $N$ do you need?

Recall Solution — L2

L2.1 First find $p$ : the fraction is $p = 120/200 = 0.6$ , and $s = 3$ . $S = \frac{1}{( 1 - 0.6 ) + \frac{0.6}{3}} = \frac{1}{0.4 + 0.2} = \frac{1}{0.6} \approx 1.667$ New time $= 200/1.667 = 120$ ms. (Sanity check the long way: unchanged part $= 80$ ms, DB part $= 120/3 = 40$ ms, total $= 80 + 40 = 120$ ms. ✓)

L2.2 As $s \to \infty$ , the term $p / s \to 0$ , so $S \to \frac{1}{1 - p} = \frac{1}{1 - 0.4} = \frac{1}{0.6} \approx 1.667$ Even making that part instant only gets you $1.667 \times$ . The other $60%$ of runtime is untouched, and it becomes the new floor.

L2.3 Rearrange $S E = p (1 - p) / N$ . Square both sides: $S E^{2} = p (1 - p) / N$ , so $N = \frac{p ( 1 - p )}{S E ^{2}} = \frac{0.20 \cdot 0.80}{0.0 1 ^{2}} = \frac{0.16}{0.0001} = 1600 samples$ At a 1 kHz sampling rate that is only $1.6$ s of profiling.

Level 3 — Analysis

Goal: decide why the profiler shows what it shows, and what to do.

L3.1 A flame graph shows handle_request (100% width) calling serialize (15%), db_query (78%), and log (7%). Inside db_query the bar is flat, wide, with no children, and CPU utilization during it is near zero. Which function should you optimize, and is the fix a code fix or an I/O fix?

L3.2 Your deterministic profiler reports that tiny_helper() has the highest self-time. But tiny_helper is a two-line function called $50$ million times. A sampling profiler ranks it far lower. Which profiler do you trust and why?

L3.3 A parent function build_report shows 95% cumulative time but only 3% self time. Should you optimize build_report itself? What does this pattern tell you?

Recall Solution — L3

L3.1 Optimize ==db_query== (78% width — it dominates). The flame graph below makes it visual: bar width is time, and db_query is the widest bar. The pattern flat + wide + no children + low CPU is the signature of waiting (I/O-bound). So the fix is an I/O fix (add a DB index, batch an N+1 query, or cache the result) — not a Python micro-optimization. Optimizing serialize (15%) would be chasing the wrong bar; by Amdahl its ceiling is $1/ (1 - 0.15) \approx 1.18 \times$ .

Figure — Performance profiling — CPU, memory, I - O profiling

L3.2 Trust the sampling profiler here. A deterministic profiler hooks every call/return, so a function called $50$ million times pays that per-call overhead $50$ million times — its measured time is inflated by the measurement itself. This distorts the ranking specifically for hot, call-heavy functions. The sampling profiler adds no per-call cost and reflects reality. See Benchmarking vs Profiling.

L3.3 No — do not optimize build_report directly. Recall from the self-vs-cumulative definition above: cumulative time = self time + all descendants. A $95%$ cumulative / $3%$ self split means build_report is a thin wrapper (a wide bar with only a sliver of self time): the real work ( $92%$ ) is burning in a callee it invokes. Descend into the children and sort by self time to find the true hotspot.

Level 4 — Synthesis

Goal: combine Amdahl, self/cumulative time, and the wall-vs-CPU split in one decision.

L4.1 A request is $500$ ms wall-clock. Profiling reveals: $50$ ms CPU parsing, $50$ ms CPU formatting, and $400$ ms waiting on 8 sequential database queries (an N+1 pattern) at $50$ ms each. You have two options:

(a) Rewrite parsing+formatting in C to make that CPU work $10 \times$ faster. (b) Batch the 8 queries into 1 query taking $50$ ms.

Compute the overall speedup and new time for each option. Which wins, and why does the CPU rewrite disappoint?

L4.2 After you apply option (b), what is the new dominant cost, and what would the next Amdahl calculation look like if you then made the remaining CPU work $10 \times$ faster?

Recall Solution — L4

Setup. Total $T = 500$ ms $= 100$ (CPU: 50 parse + 50 format) $+ 400$ (I/O wait). The figure below shows all three time-splits side by side.

Option (a) — speed CPU work $10 \times$ . The CPU part is $p_{cpu} = 100/500 = 0.2$ , with $s = 10$ : $S_{a} = \frac{1}{( 1 - 0.2 ) + \frac{0.2}{10}} = \frac{1}{0.8 + 0.02} = \frac{1}{0.82} \approx 1.220$ New time $= 500/1.220 \approx 410$ ms. Verify directly: CPU $100 \to 10$ ms, I/O stays $400$ ms, total $= 410$ ms. ✓ A huge $10 \times$ code win bought only a $\sim 1.22 \times$ request win — because the CPU was only $20%$ of the time (Amdahl's ceiling $= 1/ (1 - 0.2) = 1.25 \times$ ).

Option (b) — batch 8 queries into 1. The I/O part is $p_{io} = 400/500 = 0.8$ ; batching $8$ queries of $50$ ms into $1$ query of $50$ ms cuts $400 \to 50$ ms, i.e. speed factor $s = 400/50 = 8$ : $S_{b} = \frac{1}{( 1 - 0.8 ) + \frac{0.8}{8}} = \frac{1}{0.2 + 0.1} = \frac{1}{0.3} \approx 3.333$ New time $= 500/3.333 = 150$ ms. Verify directly: CPU $100$ ms + I/O $50$ ms $= 150$ ms. ✓

Winner: option (b), $3.33 \times$ vs $1.22 \times$ . The CPU rewrite disappoints because you optimized the small slice ( $p = 0.2$ ); Amdahl caps you no matter how brilliant the rewrite. Always attack the biggest $p$ first — which here is the I/O, exactly what the wall ≫ CPU gap told you to expect.

L4.2 After (b), new total $= 150$ ms: CPU $100$ ms + I/O $50$ ms. CPU is now the dominant cost, $p_{cpu}^{'} = 100/150 \approx 0.667$ . Making that CPU work $10 \times$ faster: $S = \frac{1}{( 1 - 0.667 ) + \frac{0.667}{10}} = \frac{1}{0.333 + 0.0667} = \frac{1}{0.4} = 2.5$ New time $= 150/2.5 = 60$ ms. The moral: the same $10 \times$ CPU rewrite that was worthless before is now worth $2.5 \times$ — because fixing the big slice changed which slice is biggest. Re-profile after every fix.

Level 5 — Mastery

Goal: derive, design, and reason about limits.

L5.1 (Derive the sample budget for a ratio of two hotspots). Two functions have true fractions $p_{1} = 0.30$ and $p_{2} = 0.25$ . You want enough samples that you can confidently distinguish which is bigger — say, the gap $p_{1} - p_{2} = 0.05$ should be at least $3$ standard errors wide. Using the worst-case error (take $p = 0.5$ , which maximizes $p (1 - p)$ , so each fraction has $S E \leq 0.25/ N$ ), find the $N$ that makes $3 \times S E_{diff} \leq 0.05$ , where the difference of two independent estimates has $S E_{diff} = S E_{1}^{2} + S E_{2}^{2}$ .

L5.2 (Design decision). You must profile a production service handling live traffic. The service is latency-sensitive: a $5 \times$ slowdown would trip its timeouts and cause an outage. Which profiler type do you deploy, and state the two properties of that profiler (from L1) that make it safe here. Then name the one thing it cannot give you that a deterministic profiler could.

Recall Solution — L5

L5.1 First, why is $p = 0.5$ the worst case? The quantity $p (1 - p)$ is a downward parabola in $p$ : it is $0$ at $p = 0$ and $p = 1$ , and symmetric about the midpoint. Its derivative is $\frac{d}{d p} p (1 - p) = 1 - 2 p$ , which is zero at $p = 0.5$ — so the maximum of $p (1 - p)$ sits at $p = 0.5$ , where $p (1 - p) = 0.25$ . Since $S E$ grows with $p (1 - p)$ , using $p = 0.5$ gives the largest (safest) error estimate, guaranteeing enough samples no matter the true $p$ .

Why the errors add in quadrature (the independence assumption). $S E_{diff} = S E_{1}^{2} + S E_{2}^{2}$ is the variance-addition rule and it holds only if the two estimates are independent. Here that is an approximation, not an exact truth: at any single sample the profiler records one running function, so the counts for func1 and func2 are actually negatively correlated (a sample credited to one cannot be credited to the other). That negative covariance would make the true $S E_{diff}$ smaller than the quadrature value. So treating them as independent is the conservative (pessimistic) choice — it slightly over-estimates the error and hence over-budgets $N$ , which is exactly what we want for a safe "how many samples is enough" bound. We proceed with the independent approximation knowing it errs on the side of caution.

Each estimate, using the worst-case $p (1 - p) = 0.25$ , has $S E = 0.25/ N$ . For the difference of two (assumed-independent) estimates the errors add in quadrature: $S E_{diff} = S E_{1}^{2} + S E_{2}^{2} = \frac{0.25}{N} + \frac{0.25}{N} = \frac{0.5}{N}$ Now require $3 \cdot S E_{diff} \leq 0.05$ and solve for $N$ step by step: $3 \frac{0.5}{N} \leq 0.05$ Divide both sides by $3$ : $\frac{0.5}{N} \leq \frac{0.05}{3} = 0.01 \overline{6}$ Square both sides (both sides positive, so the inequality direction is preserved): $\frac{0.5}{N} \leq (\frac{0.05}{3})^{2} = \frac{0.0025}{9} \approx 0.0002778$ Take reciprocals (flips the inequality) and multiply by $0.5$ : $N \geq \frac{0.5 \cdot 9}{0.0025} = \frac{4.5}{0.0025} = 1800 samples$ So $N = 1800$ samples (worst-case). At 1 kHz that is $1.8$ s of profiling to reliably tell a $30%$ hotspot from a $25%$ one. Notice the cost of resolving a small gap between two large hotspots is much higher than resolving one big hotspot alone (L2.3 needed only $1600$ for one function) — distinguishing near-ties is genuinely expensive. (And since the independence assumption over-estimated the error, $1800$ is a safe upper bound; the truly-needed count is a touch lower.)

L5.2 Deploy a sampling (statistical) profiler (e.g. py-spy, Linux perf). The two safety properties, both stated in the L1 solution:

Low overhead — it only peeks at intervals (e.g. every 1 ms) instead of hooking every call, so it does not cause the $5 \times$ slowdown that would trip timeouts. A deterministic profiler, by contrast, hooks every call/return and can slow the service 2–10×, which here means an outage — so it is disqualified.
Attaches to an already-running process without instrumenting/recompiling — you don't restart the live service or add hooks to its code; the profiler observes from outside a process that is already serving real traffic.

The one thing it cannot give: an exact call count (and exact per-call timing). Sampling is a statistical estimate — it can miss short-but-frequent functions that fall between snapshots and never reports "this ran exactly $N$ times." For exact call counts or precise per-call timing you must run a deterministic profiler in a safe offline/test environment, never on the latency-sensitive production box. In production, pair the sampling profiler with tracing/observability for the request-level picture.

Recall One-line takeaways

Amdahl's Law ranks a fix by ::: the fraction $p$ it touches, not by its local speedup. Wall ≫ CPU means the program is ::: I/O-bound (waiting), so a CPU profiler will mislead. High cumulative but low self time means the function is ::: a wrapper — the real work is in a callee. Sampling error shrinks as ::: $1/ N$ , so big hotspots resolve fast but near-ties are costly. The safe production profiler is ::: a low-overhead sampling profiler that attaches to a live process.