Building a portfolio and research roadmap: Worked examples

Definition Our reviewer model (called out up front, used everywhere)

There is no law of physics for how a reviewer reads a portfolio, so we adopt one explicit model and state it before using it — every "protocol" rule below is a consequence of this model, not a hidden assumption.

Single-impression rule (the "best-artifact headline"). A busy human forms one headline impression, and it anchors on your strongest artifact, not a sum of all of them. Why this model: attention is scarce; people remember peaks, not totals (a well-documented "peak-end" effect). So we use $max$ , not $\sum$ , for the headline.
Weighted signal score. Each artifact is scored $S = 4 D + 3 R + 2 E$ with $D$ = depth, $R$ = rigor, $E$ = explanation, each on $0 - 3$ . The weights $4, 3, 2$ encode a stated priority: depth first, rigor next, explanation last-but-real. These are modeling choices, announced here so you can argue with them.
Bounded attention. A reviewer only inspects what fits their time/click budget; anything past that budget is unseen (Row F makes this bite).

Everything downstream is arithmetic on this model. Change the model and the numbers change — but the method (turn fuzzy into checkable) does not.

The scenario matrix

Think of a matrix the way you would for signs and quadrants in trigonometry: rows are the dimension of variation, columns are the edge cases along that dimension. Every cell is a situation. Our job is to cover all of them — and below, every cell is filled by a numbered example (its tag A1, C4, … appears in the cell).

Dimension (row)	Zero / degenerate case	Typical case	Limiting / extreme case	Sign flips (good → bad)
A. Portfolio inventory	0 projects → A1	3–4 mixed artifacts → A2	40 half-finished repos → A3	quality-vs-quantity trade → A4
B. Reproduction gap	exact match, gap = 0 → B1	small gap to judge → B2	gap so large paper is unmatched → B3	your number above paper's → B4
C. Extension result	no change ( $Δ = 0$ ) → C1	modest win → C2	huge win (suspicious!) → C3	negative result ( $Δ < 0$ ) → C4
D. Roadmap time budget	0 slack (fully booked) → D1	phase fits exactly → D2	phase overruns / early → D3	phase finishes early → D4
E. Compute constraint	no GPU at all → E1	one small GPU → E2	cluster access → E3	compute wasted → E4
F. Real-world / exam twist	reviewer with 0 time → F1	admissions scoring → F2	tie-breaker at the top → F3	strong work, weak framing → F4

Definition The measurable quantities we will use

Everything below is built from a few plain-language ratios. We define each before we use it.

Reproduction gap $g$ = (paper's reported score) − (your score), in the paper's own units. It answers: "how far off am I?"
Efficiency trade ratio $T$ = (fraction of performance kept) ÷ (fraction of cost kept). It answers: "was the shortcut worth it?" We build it carefully in Row C.
Portfolio signal score $S$ = a weighted sum a reviewer forms (defined in the reviewer model above). It answers: "what does an outsider see?"

[!mistake] Two different $R$ 's? No — we renamed one. Inside the signal score $S = 4 D + 3 R + 2 E$ , the letter $R$ means rigor. The efficiency trade ratio is a completely different quantity, so to avoid shadowing we call it $T$ (for "trade"), never $R$ , everywhere on this page.

Row A — Portfolio inventory

Example A1 — the ZERO case (blank slate)

Forecast: Guess now — four things feel like more, but do they count more?

Score the one deep reproduction. Why this step? We must convert "feels impressive" into the reviewer's actual formula, not our gut. $S_{deep} = 4 (3) + 3 (2) + 2 (2) = 12 + 6 + 4 = 22.$
Score the four shallow tutorials. Why? By the single-impression rule of our reviewer model, the headline is the best artifact, not a sum — so we take the best of the four (they are identical at $S = 6$ ), not their total. $S_{shallow} = 4 (1) + 3 (0) + 2 (1) = 4 + 0 + 2 = 6.$
Compare. $22 > 6$ . Why this matters: depth is weighted $4 \times$ and rigor $3 \times$ , so shallow breadth can never catch up.

Verify: Even if you did ten shallow tutorials, the best one still scores 6, far below 22. The zero-project case is solved by going deep once, not wide. This is the concrete reason the parent note says "reproductions first."

Example A2 — the TYPICAL case (3–4 mixed artifacts)

Forecast: Does a balanced 3-artifact set beat a lone deep reproduction (which scored 22 in A1)?

Score each artifact with $S = 4 D + 3 R + 2 E$ . Why? We need the headline before the bonus. $S_{repro} = 4 (3) + 3 (2) + 2 (2) = 22, S_{ext} = 4 (2) + 3 (3) + 2 (1) = 19, S_{blog} = 4 (1) + 3 (1) + 2 (3) = 13.$
Take the best as the headline. Why? The single-impression rule anchors on your strongest work: $max (22, 19, 13) = 22$ .
Add the breadth bonus. Why this step? Three distinct pillars are present, so bonus $= 3 \times 1 = 3$ . Why a bonus at all: covering all three pillars signals a rounded researcher, which the lone-reproduction portfolio lacks. $S_{total} = 22 + 3 = 25.$

Verify: $25 > 22$ — the balanced portfolio edges out the single deep reproduction because of breadth, not despite it. This is the "three pillars" claim of the parent note, made numeric. The typical, healthy state of a portfolio is exactly this: one strong headline plus coverage.

Example A3 — the OVER-FULL / extreme case (40 half-finished repos)

Forecast: Does deleting work ever raise your average?

Average score before pruning. Why? With 40 repos the 6-click budget applies, and clicks are uniform-random (justified above: the reviewer cannot pre-sort). Each clicked repo is finished with probability $5/40$ , so the expected per-click score is $\overset{ˉ}{S}_{before} = 3 \times \frac{5}{40} = 3 \times 0.125 = 0.375.$
Average score after archiving the 35 bad ones. Why this step? Now only 5 repos exist, which is fewer than the budget of 6, so by the stated protocol the reviewer opens all 5 — and every one is good. The average is therefore $\overset{ˉ}{S}_{after} = 3 \times 1 = 3.$ (The sample size dropped from 6 to 5 because the protocol opens all repos when fewer than 6 exist — that is the stated rule, not a silent change.)
Interpret the extreme. Deleting 35 repos raised the average impression from $0.375$ to $3$ . Why? Reviewers judge by density of quality, not count.

Verify: $3/0.375 = 8 \times$ improvement from removing work. This is the arithmetic behind "archive old work" in the parent note — see also 6.5.10-Open-problemsin-AI-research for choosing what to keep working on.

Example A4 — the SIGN-FLIP cell (quality-vs-quantity trade)

Forecast: More repos feels like progress — but watch the sign.

Headline under Option Q. Why? Adding unfinished work never raises the max (single-impression rule); and the bug can only lower it. Your best repo is untouched by the new repos, so headline stays $S_{Q} = 4 (3) + 3 (2) + 2 (1) = 12 + 6 + 2 = 20,$ with a real risk of dropping if the introduced bug hits the top repo.
Headline under Option D. Why? Raising $E$ from 1 to 3 on the top repo is a $+ 2 \times 2 = + 4$ swing: $S_{D} = 4 (3) + 3 (2) + 2 (3) = 12 + 6 + 6 = 24.$
The sign flip. $S_{D} = 24 > S_{Q} = 20$ . Why "good → bad": chasing quantity (Q) at best holds you at 20 and at worst drops you, while depth (D) strictly rises. The intuitively "productive" choice is the losing one.

Verify: $24 > 20$ ; the depth swing is exactly $+ 4$ ( $S_{D} - S_{Q} = 24 - 20 = 4$ ). Quantity is a trap when quality is your bottleneck — the parent note's "narrative over chronology."

Row B — The reproduction gap

Definition The noise model behind

$z = g / σ$ (stated before we use it) A trained model's score wobbles run-to-run because of random seeds (weight init, data shuffling, dropout). We model that wobble as normally distributed noise with standard deviation $σ$ — the classic bell curve. Why normal? A score is the sum of many tiny independent random effects, and sums of many small independent things tend to a bell curve (the Central Limit Theorem). That is the assumption that makes the next tool meaningful.

The z-score $z = g / σ$ asks: "how many noise-widths is my gap?"
Why the $\pm 2 σ$ band? Under the normal model, about $95%$ of pure-noise runs land within $2 σ$ of the true value. So $∣ z ∣ < 2$ = "a gap this small happens $\geq 95%$ of the time by luck alone — don't panic"; $∣ z ∣ > 2$ = "rarer than $1$ -in- $20$ by luck — something real is going on." The $2$ is the conventional $95%$ confidence line; pick $3 σ$ if you want to be stricter. It is a choice, announced here.

The figure shows the paper's BLEU as a horizontal teal line at 27.3, a shaded teal $\pm 2 σ$ band (its numeric edges 26.7 and 27.9 marked on the vertical BLEU-score axis), and three of your runs as points: B2's 26.9 sits inside the band (safe), B3's 19.5 sits far below it (bug), and B4's 28.1 sits above it (audit). "Inside the band = explainable by luck" is the whole visual message.

Example B1 — the ZERO case (exact match, g = 0)

Forecast: Zero gap — dream come true, or too clean?

Compute the gap and its z-score. Why? Even the degenerate case must pass through the same tool (valid because the same normal noise model applies). $g = 0, z = \frac{g}{σ} = \frac{0}{0.3} = 0.$
Decide. $z = 0$ sits dead-centre of the band — on the teal line itself in the figure. Under the normal model this is the most likely single outcome, so it is a genuine match. Why still stay alert: an exactly identical number (not just within-noise) can hint you accidentally reused the paper's released checkpoint. If you trained from scratch, $g = 0$ is simply a clean pass.

Verify: $z = 0 < 2$ → match. The zero cell is the calm centre of the band; every other B example is measured as a distance from here.

Example B2 — the TYPICAL case (small gap to judge)

Forecast: Guess — is $0.4$ "close enough"?

Compute the gap. Why? We need a number before we judge it. $g = 27.3 - 26.9 = 0.4.$
Express the gap in units of noise. Why this tool? A raw gap means nothing without a scale. Dividing by $σ$ (a "z-score" — literally "how many noise-widths away", meaningful because the noise is normal) answers "could randomness alone explain this?" $z = \frac{g}{σ} = \frac{0.4}{0.3} \approx 1.33.$
Decide. $z = 1.33$ is inside the $\pm 2$ band; the figure's B2 point lands inside the teal shading. Verdict: probable match, no bug hunt needed. Why: chasing a within-noise gap wastes weeks.

Verify: $z = 1.33 < 2$ , so the point is inside the $2 σ$ band $[27.3 - 0.6, 27.3 + 0.6] = [26.7, 27.9]$ , and $26.9$ is inside. Report it honestly in your RESULTS.md.

Example B3 — the EXTREME case (gap so large the paper is unmatched)

Forecast: Bug or bad luck?

Gap and z-score. Why? Same measurement, new number. $g = 7.8, z = \frac{7.8}{0.3} = 26.$
Decide. $z = 26 ≫ 2$ — see how far below the band the figure's lowest point sits. Under the normal model a $26 σ$ deviation has probability essentially zero, so randomness is impossible here. Why this matters: this is a structural bug, exactly the "forgot the causal mask" story in the parent note.

Verify: $26 > 2$ by a huge margin → hunt the bug. The lesson: the same verification tool ( $z = g / σ$ ) resolves the calm zero cell, the small gap, and this extreme; only the answer changes.

Example B4 — the SIGN-FLIP case (you beat the paper)

Forecast: Is beating the paper good news?

Signed gap. Why keep the sign? A negative $g$ means you exceeded the paper — the direction carries information. $g = 27.3 - 28.1 = - 0.8, ∣ z ∣ = \frac{0.8}{0.3} \approx 2.67.$
Decide. $∣ z ∣ = 2.67 > 2$ : too far outside the $95%$ band to be a seed fluctuation, and it sits above the band in the figure. Why care? Beating a paper by $> 2 σ$ usually means an evaluation leak (e.g., you accidentally tested on training data). Audit before you brag.

Verify: $∣ - 0.8 ∣/0.3 = 2.67 > 2$ → investigate. A too-large positive surprise is as suspicious as a too-large negative one — both live outside the band.

Row C — Extension results (building the ratio $T$ )

The figure plots your extension in a $q$ – $p$ plane: horizontal axis $q$ = fraction of cost kept, vertical axis $p$ = fraction of performance kept. The dashed diagonal $p = q$ is the break-even line $T = 1$ ; the shaded region above it is $T > 1$ ("worth it"). C2 sits well above the line (strong win), C1 sits exactly on it (no change), and C4 sits just barely above it (marginal negative-result case).

Example C1 — the ZERO case (Δ = 0, no change)

Forecast: Nothing changed — is that a failure?

Compute $T$ . Why? Same tool, degenerate inputs. $p = \frac{92.3}{92.3} = 1, q = \frac{0.8}{0.8} = 1, T = \frac{1}{1} = 1.$
Interpret the break-even. $T = 1$ exactly — the point lands on the dashed line in the figure, and $Δ = 0$ . Why this matters: an unchanged result is a valid negative ("this optimizer swap does nothing here") and is worth one honest paragraph, never a headline.

Verify: $T = 1$ and $Δ = 0$ → report as a null result, not a win. Zero change is a real cell of the matrix, not a bug.

Example C2 — the TYPICAL WIN case

Forecast: $0.6%$ accuracy lost for $37.5%$ compute saved — good deal?

Performance kept. Why? First ratio of $T$ . $p = \frac{91.7}{92.3} = 0.99350.$
Cost kept. Why? Second ratio. $q = \frac{0.5}{0.8} = 0.625.$
Trade ratio. Why the division? To collapse the trade into one verdict number. $T = \frac{0.99350}{0.625} = 1.590.$
Interpret. $T = 1.59 > 1$ : you kept $59%$ more of the performance-share than of the cost-share. Strong, publishable extension. See the figure — the point sits well above the $T = 1$ break-even line.

Verify: $T \approx 1.59 > 1$ . Also sanity-check the parent's claim " $37.5%$ compute savings": $1 - q = 1 - 0.625 = 0.375 = 37.5%$ . ✓ Units: $p, q, T$ are all dimensionless (ratios).

Example C3 — the EXTREME case (huge win, Δ ≫ 0 — be suspicious)

Forecast: Massive accuracy gain and 4× cheaper — dream, or red flag?

Compute $T$ . Why? Quantify the "too good." $p = \frac{99.0}{92.3} = 1.07259, q = \frac{0.2}{0.8} = 0.25, T = \frac{1.07259}{0.25} = 4.2904.$
Interpret the magnitude. $T \approx 4.29$ with $Δ = + 6.7$ — the point would sit far up in the top-left of the figure, way above break-even. Why be suspicious: a $6.7$ -point accuracy jump while getting $4 \times$ cheaper contradicts the usual accuracy–cost frontier. Such a result almost always means a leak or metric bug (train/test overlap, evaluating on the wrong split).
Action. Re-run with a held-out set and a fixed seed before claiming anything. Extraordinary $T$ demands extraordinary checking — link the audit trail into 6.5.1-Interpretability-and-explainability to show why it improved, or admit the bug.

Verify: $T \approx 4.29 ≫ 1$ and $Δ = 99.0 - 92.3 = 6.7 > 0$ → treat as suspect, not triumph. The extreme-win cell is where honesty matters most.

Example C4 — the SIGN-FLIP case (negative result, Δ < 0)

Forecast: You lost accuracy and barely saved compute — is this worthless?

Ratios. Why? Same machinery. $p = \frac{88.0}{92.3} = 0.95341, q = \frac{0.7}{0.8} = 0.875.$
Trade ratio. Why? Verdict. $T = \frac{0.95341}{0.875} = 1.0896.$
Interpret. $Δ = - 4.3$ (accuracy fell), yet $T \approx 1.09 > 1$ technically — but only barely, because you paid a $4.3$ -point accuracy hit for a mere $12.5%$ compute cut. This is the "barely worth it, probably not" cell. A good write-up says why the long-range dependencies mattered here (link to 6.5.1-Interpretability-and-explainability to actually visualise what got lost).

Verify: $T \approx 1.09$ , $Δ = - 4.3$ , and compute cut $1 - 0.875 = 0.125 = 12.5%$ . A negative- $Δ$ result with $T$ barely above 1 is exactly the kind of honest science the parent note praises.

Row D — Roadmap time budget

Example D1 — the ZERO-SLACK case (fully booked)

Forecast: No slack — how bad is one surprise?

Slack available. Why? Slack is planned time minus committed time. $slack = 12 - 12 = 0.$
Effect of the shock. Why this step? Any overrun with zero slack pushes the deadline one-for-one: $finish week = 12 + 1 = 13 \Rightarrow 1 week late.$
Lesson. A zero-slack plan has no absorption capacity — the first surprise slips you. Why it matters: always reserve buffer (see D2).

Verify: $12 - 12 = 0$ slack, so a 1-week shock → 13-week finish, exactly 1 week over. The degenerate "fully booked" plan is fragile by construction.

Example D2 — the TYPICAL case (phase fits exactly)

Forecast: Does a plan with a buffer actually finish inside the window?

Sum committed time. Why? Compare against the 12-week window. $6 + 5 = 11 weeks of committed work.$
Check against the window with buffer. Why this step? The 1-week buffer sits on top: $11 + 1 = 12 \leq 12$ . $used = 11 \leq 12 = window, slack left = 12 - 11 = 1.$
Interpret. The phase fits with exactly 1 week of unused buffer — the healthy, typical outcome. Why it matters: a plan that fits and keeps buffer is what "success metric on time" looks like.

Verify: $6 + 5 = 11 \leq 12$ and slack $= 1 \geq 0$ → fits on time. This typical cell is the target every roadmap phase should aim for.

Example D3 — the EXTREME case (overrun via Parkinson's Law)

Forecast: 8 of 12 weeks gone — comfortable or doomed?

Time left. Why? The denominator of the whole question. $t_{left} = 12 - 8 = 4 weeks .$
Current rate. Why? To project forward we need a velocity. You did the skills (6 weeks) plus $60%$ of the repro, so effective repro time spent $= 8 - 6 = 2$ weeks for $60%$ . $rate = \frac{0.60}{2} = 0.30 of the reproduction per week .$
Weeks needed for the last 40%. Why divide? Remaining work ÷ velocity = time. $t_{need} = \frac{0.40}{0.30} = 1.333 weeks .$
Compare. $t_{need} = 1.33 \leq 4 = t_{left} \Rightarrow$ on track, slack $= 4 - 1.33 = 2.67$ weeks.

Verify: $1.33 \leq 4$ ✓. The slack ( $2.67$ weeks) is your buffer against Parkinson's Law — time-box the write-up into it so the work does not expand to fill it.

Example D4 — the SIGN-FLIP case (finish early — what to do)

Forecast: Rush ahead, or polish?

Value of Option X over 3 weeks. Why? Baseline plan. $V_{X} = 3 \times 3 = 9.$
Value of Option Y over 3 weeks. Why? It front-loads value then flatlines. $V_{Y} = 5 \times 2 + 0 \times 1 = 10.$
Compare. $V_{Y} = 10 > 9 = V_{X}$ . Why: a finished, framed artifact (Y) out-values a rushed head start (X). Slack should first buy polish, then progress.

Verify: $10 > 9$ → polish the current artifact before racing ahead. Finishing early is a "good" surprise that still needs a decision.

Row E — Compute constraint

Example E1 — the ZERO-GPU degenerate case

Forecast: Is a smaller-scale experiment "real" research?

Feasibility check. Why? Confirm it fits the hard budget. $5 GPU-hours \leq 5 budget \Rightarrow feasible.$
Expected information value. Why? A proxy that agrees $85%$ of the time still resolves most comparisons; the null (no experiment) is a coin flip at $50%$ : $gain = 0.85 - 0.50 = 0.35.$
Decide. Positive gain, within budget → run it, and state the proxy caveat. Why: an honest small-scale result beats a grand experiment you cannot run. See 6.5.3-Federated-learning for another "compute-is-elsewhere" mindset.

Verify: $5 \leq 5$ ✓ and $0.85 > 0.50$ ✓ → the degenerate no-GPU case still yields publishable evidence.

Example E2 — the TYPICAL case (one small GPU)

Forecast: 8 configs on a modest GPU — one week enough?

Total cost of the sweep. Why? Compare against the weekly budget. $8 \times 6 = 48 GPU-hours .$
Compare to budget. Why this step? $48 > 40$ — it does not fit in one week. $48 - 40 = 8 GPU-hours over.$
Fix. Why? Drop the 2 least-promising configs ( $6 \times 2 = 12$ saved): $48 - 12 = 36 \leq 40$ , leaving $40 - 36 = 4$ hours headroom.

Verify: $8 \times 6 = 48 > 40$ (over by 8); after cutting 2 configs, $36 \leq 40$ with 4 hours spare. The typical single-GPU life is prioritise, then sweep.

Example E3 — the EXTREME case (cluster access)

Forecast: 64 GPUs — surely 64× faster?

Split the work. Why? Only the parallel part shrinks with more GPUs (this is Amdahl's Law). $serial = 0.20 \times 20 = 4 GPU-hours, parallel = 0.80 \times 20 = 16 GPU-hours .$
Wall-clock time on 64 GPUs. Why divide only the parallel part? The serial 4 hours stay fixed; the 16 parallel hours split across 64 GPUs: $t = 4 + \frac{16}{64} = 4 + 0.25 = 4.25 hours .$
Real speedup. Why? Compare to the 20-hour single-GPU run: $speedup = \frac{20}{4.25} \approx 4.71 \times .$

Verify: $t = 4.25$ hours and speedup $\approx 4.71 \neq = 64$ . The extreme "infinite compute" cell is bottlenecked by the serial fraction — throwing GPUs at it saturates fast.

Example E4 — the SIGN-FLIP case (compute wasted on the wrong thing)

Forecast: More tuning hours feels better — does it help here?

Value of Plan A. Why? Invalid results are worth nothing no matter the compute. $V_{A} = 100 \times 0 = 0.$
Value of Plan B. Why? Fix first (0 value for 10 h), then valid tuning: $V_{B} = 90 \times 1 = 90.$
The sign flip. $V_{B} = 90 ≫ V_{A} = 0$ . Why "good → bad": pouring compute into a broken setup is negative-productivity — busy but worthless. Correctness gates value.

Verify: $V_{A} = 0 < 90 = V_{B}$ → fix the pipeline before spending compute. Wasted compute is the "good intentions, bad outcome" cell of Row E.

Row F — Real-world / exam twist

Example F1 — the ZERO-TIME reviewer (degenerate)

Forecast: With no scrolling, which number does the reviewer see?

Identify what is read. Why? With zero scroll time, only the first item counts. $impression = S_{first} = 13.$
Compare to your best. Why this step? Your strongest work (22) is below the fold and never seen — a $22 - 13 = 9$ -point impression lost purely to ordering.
Fix. Why? Reorder so the best project leads: impression jumps from 13 to 22 at zero cost.

Verify: Impression $= 13$ when best (22) is buried; reordering recovers the full 9-point gap. The zero-time reviewer is why the parent note says "lead with your best work."

Worked examples — Building a portfolio and research roadmap