Code-generation agents: Exercises — AI-ML deep dive

Before we start, one reused tool, stated from zero so nobody is stuck:

Intuition Reading figure s01

The horizontal axis is $k$ , the number of attempts; the vertical axis is the success probability $1 - (1 - p)^{k}$ . Three chalk curves are drawn, one per single-try skill $p$ . Follow the pale-yellow curve ( $p = 0.4$ ): the yellow labels on it read $40%$ at $k = 1$ , jump to $78%$ at $k = 3$ , and reach $92%$ at $k = 5$ — the exact numbers Exercises 2.1–2.2 ask about. The dashed white line marks the $95%$ target; notice the yellow curve only crosses it past $k = 5$ , which is precisely why Exercise 2.2's answer is $k = 6$ . The steepness on the left of every curve and its flattening on the right is the visual signature of diminishing returns: the first extra attempt buys the most probability, each later one buys less. That single shape is the whole argument for why agents loop instead of answering once.

Level 1 — Recognition

Exercise 1.1 (L1)

Which single property most sharply separates a code-generation agent from a plain code-completion model?

Recall Solution

The feedback loop with the environment. A completion model does one-shot generation (predict the next tokens, done). An agent runs its code, reads outputs and errors, and feeds that back in — the sense–decide–act loop from Tool use in AI systems. Everything else (bigger model, more tokens) is a difference of degree; the loop is a difference of kind.

Exercise 1.2 (L1)

Match each of the four agent components to its one-line job: Task understanding, Code synthesis, Execution & verification, Iterative refinement.

Recall Solution

Task understanding ::: turn natural-language requirements into structured goals.
Code synthesis ::: produce syntactically and semantically correct code.
Execution & verification ::: run code in a sandbox, capture outputs and errors.
Iterative refinement ::: debug failures, adapt to feedback, converge to a working solution.

Exercise 1.3 (L1)

The agent keeps a memory of everything it has tried, called the execution history and written $H_{t}$ — a growing list where each row is one triple $(A_{t}, o_{t}, e_{t})$ : the action it took, the output it saw, and the error it hit at step $t$ . At the next step this memory grows by one row, following the state-update rule $H_{t + 1} = H_{t} \cup {(A_{t}, o_{t}, e_{t})},$ where $\cup$ (say "union") just means "add this new row to the set we already had." Name what each of the three stored things inside a row is.

Recall Solution

$H_{t}$ ::: the execution history at step $t$ — the full list of past $(A, o, e)$ rows; it is the agent's memory.
$A_{t}$ ::: the action the LM produced at step $t$ (usually a code snippet or edit).
$o_{t}$ ::: the output the sandbox printed (stdout).
$e_{t}$ ::: the error trace, if any (stderr / exception). Together the three form one row of $H_{t}$ — "what I tried and what happened."

Level 2 — Application

Exercise 2.1 (L2)

A code agent has single-attempt success rate $p = 0.4$ . Compute the probability of solving the task in 3 or fewer attempts.

Recall Solution

Use $1 - (1 - p)^{k}$ with $p = 0.4$ , $k = 3$ : $1 - (1 - 0.4)^{3} = 1 - 0. 6^{3} = 1 - 0.216 = 0.784 = 78.4%.$ What it looks like: on the s01 curve, walk along the pale-yellow ( $p = 0.4$ ) line from $k = 1$ (its $40%$ label) up to $k = 3$ (its $78%$ label) — that visible jump is a single agent given three tries.

Exercise 2.2 (L2)

Same agent, but now with $p = 0.4$ . How many attempts $k$ are needed so that success probability first reaches at least 95%?

Recall Solution

Solve $1 - 0. 6^{k} \geq 0.95 \Rightarrow 0. 6^{k} \leq 0.05$ . We need to pull $k$ down from the exponent. A logarithm is exactly the tool for that, and here is why it works as a picture: think of $0. 6^{k}$ as starting at $1$ and shrinking by a factor $0.6$ at each step — a chain of equal-ratio shrinks. Plot those values on a ruler whose tick marks are spaced by multiplication (a "log ruler," like a piano where each octave doubles): equal multiplicative steps become equally-spaced marks. On that ruler, "multiply by $0.6$ , $k$ times" turns into "take $k$ equal steps," i.e. repeated multiplication becomes repeated addition of step-sizes. The size of one step is $ln 0.6$ and the total distance to travel is $ln 0.05$ , so the number of steps is their ratio: $k \geq \frac{l n 0.05}{l n 0.6} = \frac{- 2.9957}{- 0.5108} \approx 5.86.$ Since $k$ must be a whole number of attempts, round up: $k = 6$ . Check: $1 - 0. 6^{6} = 1 - 0.046656 = 0.9533 \geq 0.95$ . ✓ (And $k = 5$ gives $0.9222 < 0.95$ , so $6$ is the first that works — matching where the yellow curve crosses the dashed $95%$ line in s01.)

Exercise 2.3 (L2)

For the prime-checker, why does the loop stop at $n$ instead of $n$ ? Verify the reasoning for $n = 36$ .

Recall Solution

If $n = a \times b$ with both $a > n$ and $b > n$ , then $a \times b > n \cdot n = n$ — a contradiction. So any factor pair must contain a member $\leq n$ ; checking up to $n$ can never miss a factor. For $n = 36$ : $36 = 6$ . Factors found by checking $2..6$ : $2$ (since $36%2 = 0$ ). We catch it immediately, never needing to test $9, 12, 18$ — their partners $4, 3, 2$ are all $\leq 6$ .

Exercise 2.4 (L2)

Compute $pass@ 2$ for a task where you drew $n = 5$ samples and $c = 2$ were correct.

Recall Solution

$pass@ 2 = 1 - \frac{( 2 5 - 2 )}{( 2 5 )} = 1 - \frac{( 2 3 )}{( 2 5 )} = 1 - \frac{3}{10} = 0.7 = 70%.$ Meaning: if you randomly grab $2$ of your $5$ generations, there is a $70%$ chance at least one is correct.

Level 3 — Analysis

Exercise 3.1 (L3)

An agent ranks files in a codebase against a search query using a hybrid retrieval score. Here $f$ stands for a candidate file (its code text) and $q$ for the query the user typed (e.g. "find login function"); $e_{f}$ and $e_{q}$ are their embedding vectors (numeric summaries of meaning). The score is $score (f, q) = BM25 (f, q) + α \cdot embedding_sim (e_{f}, e_{q}),$ where BM25 measures keyword overlap between file $f$ and query $q$ , embedding_sim measures meaning-similarity, and $α$ balances the two. Explain, with a concrete example, why a keyword-only search ( $α = 0$ ) can miss the right file, and why an embedding-only search can also miss.

Recall Solution

Keyword-only miss ( $α = 0$ ): query $q =$ "find login function." The real handler file $f$ defines authenticate_user(). BM25 scores on shared tokens; "login" never appears in $f$ , so it ranks low — a synonym gap.
Embedding-only miss: query "call process_v2" where two files are semantically about "processing." Embeddings judge them nearly equal and may rank the wrong one first; they blur the exact identifier that a keyword match would nail. Combining them (Retrieval-Augmented Generation style) lets exact names and meaning both vote. $α$ tunes how loud the semantic voter is.

Exercise 3.2 (L3)

An agent hits IndexError: list index out of range and immediately wraps every list access in a bounds check. The error disappears but the output is silently wrong. Diagnose what really happened and give the correct debugging order.

Recall Solution

The bounds check masked a symptom without fixing the cause. The true bug was almost certainly an off-by-one or a wrong loop bound producing a bad index; by clamping the access, the agent now reads/writes the wrong element instead of crashing — worse, because it fails silently. Correct order (from the parent's error-recovery section):

Print the list length and the offending index.
Inspect the loop's termination condition (range(...) bounds).
Verify counter initialization.
Only then, add a defensive check if the logic genuinely needs one. This is reasoning about data flow, not pattern-matching the error keyword.

Exercise 3.3 (L3)

Two agents solve the same benchmark. Agent A: $p = 0.30$ per attempt, allowed $k = 10$ attempts. Agent B: $p = 0.55$ per attempt, allowed $k = 3$ attempts. Which has the higher chance of eventually succeeding? Compute both.

Recall Solution

Agent A: $1 - (1 - 0.30)^{10} = 1 - 0. 7^{10} = 1 - 0.0282475 = 0.9718 \approx 97.2%$ .
Agent B: $1 - (1 - 0.55)^{3} = 1 - 0.4 5^{3} = 1 - 0.091125 = 0.9089 \approx 90.9%$ . Agent A wins (97.2% vs 90.9%), despite a much weaker single attempt. The lesson: iteration budget compounds — a mediocre agent that gets many tries can beat a strong agent that gets few. This is exactly why the loop matters.

Level 4 — Synthesis

Exercise 4.1 (L4)

Design a test-first agent workflow for the task "write clamp(x, lo, hi) that returns x bounded into [lo, hi]." Write the tests the agent should generate (covering edge cases), then the implementation, and state which case each test defends.

Recall Solution

Step 1 — tests (formalize the spec, expose edge cases):

assert clamp(5, 0, 10) == 5    # inside range -> unchanged
assert clamp(-3, 0, 10) == 0   # below lo -> snaps to lo
assert clamp(42, 0, 10) == 10  # above hi -> snaps to hi
assert clamp(0, 0, 10) == 0    # on the lower boundary
assert clamp(10, 0, 10) == 10  # on the upper boundary
assert clamp(5, 5, 5) == 5     # degenerate lo == hi

Step 2 — implementation:

def clamp(x, lo, hi):
    if x < lo:
        return lo
    if x > hi:
        return hi
    return x

Step 3 — case coverage: the six tests defend, in order: interior value, below-range, above-range, lower boundary (inclusive), upper boundary (inclusive), and the degenerate $l o = hi$ collapse. Boundaries and the degenerate case are exactly where off-by-one and < vs <= bugs live — writing tests first forces the agent to name them before coding, per Automated testing discipline.

Exercise 4.2 (L4)

Sketch (in words + a small mermaid diagram) a multi-agent pipeline for building a small REST endpoint, and justify one place where parallelism is safe and one place where it is not.

Recall Solution

flowchart LR
  A["Architect agent"] -->|plan| B["Implementation agent 1"]
  A -->|plan| C["Implementation agent 2"]
  B -->|code| D["Testing agent"]
  C -->|code| D
  D -->|results| E["Review agent"]
  E -->|approve or revise| A

Safe parallelism: two implementation agents writing independent modules (e.g., a data model file and a route handler file) can run at once — they touch disjoint files in the shared workspace.
Unsafe parallelism: the architect deciding the shared interface (function signatures both modules import). If two agents each invent the signature in parallel, they diverge and the code will not link. This must be a serial, single-source-of-truth step before fan-out.

Level 5 — Mastery

Exercise 5.1 (L5)

An agent samples code independently at $p = 0.4$ per attempt, but each failed attempt costs $1$ compute unit and each success ends the loop. You have a hard budget of $k = 4$ attempts. (a) What is the probability of success within budget? (b) What is the expected number of attempts consumed (counting the successful one, and counting all $4$ if it never succeeds)?

Recall Solution

(a) $P (success) = 1 - (1 - 0.4)^{4} = 1 - 0. 6^{4} = 1 - 0.1296 = 0.8704 = 87.04%.$

(b) Let $q = 1 - p = 0.6$ . First, why the summation looks the way it does. An expectation is "each possible value of $N$ , weighted by how likely that value is, all added up" — a weighted average. So we list every outcome, its probability, and its cost, then sum probability × cost.

The loop reaches attempt $i$ only if the first $i - 1$ attempts all failed (probability $q^{i - 1}$ ) and then attempt $i$ succeeds (probability $p$ ). That outcome consumes exactly $i$ attempts. So its contribution is $i q^{i - 1} p$ . This is a truncated geometric distribution — "geometric" because each extra failure multiplies in another factor $q$ ; "truncated" because we cut it off at the budget $k = 4$ .
The one leftover outcome is "never succeeded in $4$ tries," probability $q^{4}$ , which still consumes all $4$ attempts — contributing $4 q^{4}$ . Adding the success-outcomes plus the never-succeed outcome gives the whole distribution: $E [N] = succeeds on attempt i i = 1 \sum 4 i q^{i - 1} p + never succeeds 4 q^{4} .$ Term by term with $p = 0.4, q = 0.6$ :
$i = 1$ : $1 \cdot 0.4 = 0.4$
$i = 2$ : $2 \cdot 0.6 \cdot 0.4 = 0.48$
$i = 3$ : $3 \cdot 0.36 \cdot 0.4 = 0.432$
$i = 4$ : $4 \cdot 0.216 \cdot 0.4 = 0.3456$
never: $4 \cdot 0.1296 = 0.5184$ $E [N] = 0.4 + 0.48 + 0.432 + 0.3456 + 0.5184 = 2.176 attempts .$ Interpretation: even with a 4-attempt cap, you expect to spend only ~2.18 attempts, because most runs finish early. This is the compute-cost side of the iteration story: high success and modest average cost.

Exercise 5.2 (L5)

You can generate $n = 6$ samples for a task, of which $c = 3$ turn out correct. Compare $pass@ 1$ , $pass@ 2$ , and $pass@ 3$ , and explain the shape of the trend.

Recall Solution

With $n = 6, c = 3$ , non-correct pool $= 3$ :

$pass@ 1 = 1 - \frac{( 1 3 )}{( 1 6 )} = 1 - \frac{3}{6} = 0.5000$ .
$pass@ 2 = 1 - \frac{( 2 3 )}{( 2 6 )} = 1 - \frac{3}{15} = 0.8000$ .
$pass@ 3 = 1 - \frac{( 3 3 )}{( 3 6 )} = 1 - \frac{1}{20} = 0.9500$ . Trend: $50% \to 80% \to 95%$ — rising and concave (each extra pick adds less than the previous, because it gets harder to draw an all-wrong subset once the pool of wrong ones is small). This mirrors the diminishing-returns curve on figure s01: the first extra try buys the most.

The bars above show the 5.2 result: notice the shrinking gaps between $k = 1, 2, 3$ — concavity you can see.

Recall Self-check: could you re-derive these from scratch?

One-attempt-vs-k relationship ::: $1 - (1 - p)^{k}$ , because all-fail is $(1 - p)^{k}$ and success is its complement. Why $n$ bounds the prime loop ::: any factor pair has a member $\leq n$ , else the product exceeds $n$ . Why pass@k uses combinations not powers ::: it draws from a fixed pool of $n$ samples without replacement.

Parent: Code-generation agents · related tools: Program synthesis, Prompt engineering.