Code-generation agents: Question bank — AI-ML deep dive

First let us pin down the words and symbols this page reuses, so nothing is used before it is defined.

Definition Vocabulary used below

One-shot generation: the model writes code once, and whatever it produces is the final answer — no running, no feedback.
The agent loop: write code → run it in a sandbox → read the output/error → decide the next edit → repeat. This feedback with the environment is what makes it an agent.
Sandbox: an isolated place to execute code so its output and errors can be captured without touching the real system.
Independent attempts: two tries where the second learns nothing from the first — a fresh dice roll each time.
TDD (Test-Driven Development): a workflow where you write the tests first, then write code whose only job is to make those tests pass.
pass@k: the fraction of problems for which at least one of $k$ sampled solutions is correct.
$p$ : the single-attempt success rate — the probability that one independent try produces correct code. It is one number between 0 and 1.

Definition Loop symbols (state, action, history)

The agent runs in discrete steps counted by a time index $t = 0, 1, 2, \dots$ — think of $t$ as "which round of the loop we are in."

$R$ = the user requirements (what was asked), fixed for the whole task.
State: everything the agent knows right now. The parts that change are the current code $C_{t}$ and the history $H_{t}$ .
$C_{t}$ = the code as it stands at step $t$ (the partial or full program).
Action $A_{t}$ = the thing the agent decides to do at step $t$ : generate new code, modify existing code, ask for clarification, or declare done. State is "what is", action is "what to do next".
Execution outputs: running an action gives $o_{t}$ = the output (what the program printed) and $e_{t}$ = the error trace (empty if it ran cleanly).
$H_{t}$ = the history, the running log of past triples ${(A_{i}, o_{i}, e_{i})}_{i < t}$ — every action tried and what happened.
$P_{t}$ = the prompt built at step $t$ : $P_{t} = [R, C_{t}, H_{t}, system instructions]$ . The update rules: $C_{t + 1} = Apply (C_{t}, A_{t})$ (edit the code) and $H_{t + 1} = H_{t} \cup {(A_{t}, o_{t}, e_{t})}$ (append the new record).

Two more clusters of symbols appear in the traps below — the ones behind the pass@k metric and the ones behind hybrid retrieval. Let us earn them before we use them.

Definition pass@k symbols and the binomial coefficient

To estimate pass@k in practice you generate a pool of candidate solutions for one problem and check each:

$n$ = the total number of samples you generated for that problem.
$c$ = how many of those $n$ samples are correct. So $0 \leq c \leq n$ .
$k$ = how many samples you are allowed to draw and submit. This is a whole number with $k \geq 1$ (you must draw at least one) and, for the standard formula, $k \leq n$ (you cannot draw more than you generated).

The binomial coefficient $(b a)$ , read " $a$ choose $b$ ", counts how many different ways you can pick $b$ items out of $a$ , order not mattering. For example $(2 4) = 6$ : from four items there are six two-item groups. By convention $(b a) = 0$ whenever $b > a$ (you cannot pick more items than exist) and $(0 a) = 1$ (there is exactly one way to pick nothing).

Formula Why pass@k is a hypergeometric count

pass@k = "if I draw $k$ of the $n$ samples, chance that at least one is correct." Compute the complement — the chance all $k$ drawn samples are from the $n - c$ wrong ones — because "at least one correct" and "none correct" are opposites that add to 1.

Total ways to choose $k$ samples out of $n$ : $(k n)$ (the denominator — all equally likely draws).
Ways to choose $k$ samples that are all wrong (only from the $n - c$ incorrect ones): $(k n - c)$ (the numerator).
So $Pr (none correct) = \frac{( k n - c )}{( k n )}$ , and therefore $pass@ k = 1 - \frac{( k n - c )}{( k n )}$ This is drawing without replacement — each sample is taken from a fixed pool, unlike $1 - (1 - p)^{k}$ which re-flips the same coin. Averaging this over all problems in a benchmark gives the reported pass@k.

Definition Retrieval symbols: BM25, embeddings, and

$α$ When an agent searches a codebase for a query $q$ (like "find login function"), it scores each file $f$ :

BM25 $(f, q)$ = a classic keyword score: high when the file literally contains the query's words (exact function names, identifiers).
embedding_sim $(e_{f}, e_{q})$ = a meaning score: each of the file and the query is turned into a vector ( $e_{f}, e_{q}$ ), and their similarity is high when they are about the same idea even with different words ("authenticate" ≈ "login").
$α$ = the balance parameter, a chosen non-negative number that says how much weight the meaning score gets relative to the keyword score. Small $α$ leans on exact keywords; large $α$ leans on semantics. The combined score is $score (f, q) = BM25 (f, q) + α \cdot embedding_sim (e_{f}, e_{q})$ .

Now walk the three figures — each one carries an idea you will be quizzed on.

Figure 1 draws one turn of the agent loop. Follow the arrows clockwise from the cyan State box: the state $(C_{t}, H_{t})$ feeds into building the prompt $P_{t}$ , which the LLM turns into an action $A_{t}$ (amber), which runs in the sandbox to produce $o_{t}, e_{t}$ , which updates the state — and the amber return arrow closes the loop into the next round. The trap "sandbox is only about security" dies here: the sandbox is the box that produces the feedback.

Figure 2 plots $1 - (1 - p)^{k}$ as $k$ grows, for $p = 0.4$ (cyan) and $p = 0.7$ (amber). Notice both curves climb toward the dashed line at 1 but never cross it — this is exactly why " $3 \times 0.4 = 1.2$ , guaranteed" is nonsense: probability is capped at 1. The annotated point confirms $p = 0.4, k = 3$ lands at $0.784$ , not $1.2$ .

Figure 3 shows why retrieval is hybrid. For the query "find login function", auth.py wins on the cyan BM25 bar (exact keyword hit) while login_view.py wins on the amber embedding bar (semantic match). Neither alone ranks both correctly; the white hybrid bar (with $α = 0.6$ ) lifts both to the top. This is the trap "embeddings alone are enough" made visible.

True or false — justify

TDD in an agent means the agent writes tests before it writes the implementation.

True — the agent generates tests first so the specification is made concrete and machine-checkable, then writes code aimed at making those tests pass. Tests-after would only confirm whatever the code already happens to do.

A code-generation agent is just a bigger code-completion model.

False — completion predicts the next tokens inside a file; an agent adds a loop that runs code, reads errors, and edits again. The defining difference is feedback with the environment, not model size.

If single-try success is

p = 0.4

, three independent tries give

3 \times 0.4 = 1.2

, i.e. guaranteed success.

False — you cannot add probabilities like that; the sum

1.2

is bigger than

1

, which is impossible for a probability. The correct rule multiplies failures:

1 - (1 - p)^{k} = 1 - 0. 6^{3} = 0.784

, about

78.4%

, never more than 1.

pass@k and

1 - (1 - p)^{k}

are the same formula.

False —

1 - (1 - p)^{k}

assumes every attempt has the same independent success rate

p

. pass@k estimates from a finite pool of

n

samples containing

c

correct ones using the hypergeometric count

(k n - c) / (k n)

, i.e. drawing without replacement from what was actually generated.

Running generated code in a sandbox is only about security.

False — security (isolation) matters, but the engine of the agent loop is that execution produces the output

o_{t}

and error

e_{t}

the agent needs to decide its next edit. Without execution there is no feedback.

Hybrid retrieval (BM25 + embeddings) is redundant because embeddings already capture meaning.

False — embeddings catch semantic similarity ("authenticate" ≈ "login") but can miss an exact token like a specific function name; BM25 nails exact keywords but is blind to synonyms. Each covers the other's blind spot.

More iterations always increase the chance of success.

Mostly true if attempts are genuinely independent — but real iterations aren't independent, so if the agent keeps re-reading the same misleading error it can loop on the same wrong fix and never improve. The formula's monotonic climb assumes fresh, uncorrelated tries.

Multi-agent code systems are better because more agents means more compute.

False — the benefit is specialization and parallelism (an architect, implementers, a tester each optimized for one job), not raw compute. Poorly coordinated agents can duplicate work or overwrite each other's files.

Spot the error

"The agent's action

A_{t}

can only ever be generate new code."

Wrong —

A_{t}

is a choice among several: generate new code, modify existing code, request clarification, or declare the task complete. Collapsing it to one option ignores the planning step.

"State update just replaces the code:

C_{t + 1} = A_{t}

Wrong — the update is

C_{t + 1} = Apply (C_{t}, A_{t})

, an edit applied to the existing code, and the history

H_{t + 1} = H_{t} \cup {(A_{t}, o_{t}, e_{t})}

also grows. The past is appended, not thrown away.

"For is_prime(n) we must test every divisor from 2 up to

n - 1

Wrong — checking up to

n

suffices, because if

n = a \times b

with

a > n

then

b < n

and would already have been caught. Testing to

n - 1

is correct but wastefully slow.

"An IndexError: list index out of range is fixed by adding a bounds check where it crashed."

Wrong in general — the crash site is often not the cause. The real bug is usually an off-by-one loop bound or a bad counter set earlier; a blind bounds check hides the symptom while the wrong logic remains.

"pass@1 measures whether the model ever solves the problem."

Wrong — pass@1 is the chance a single sample is correct; "ever solves it" is pass@k for larger

k

. Conflating them makes an iterating agent look worse than it is.

"The context prompt

P_{t}

should contain only the current code

C_{t}

Wrong —

P_{t} = [R, C_{t}, H_{t}, system instructions]

. Dropping the requirements

R

or the execution history

H_{t}

means the model forgets what was asked and what already failed.

Why questions

Why do agents iterate at all instead of just generating once?

Because first-try success on real code is low (~30–70%); syntax slips, logic bugs, and edge cases only surface when the code runs, so a feedback loop is needed to catch and repair them.

Why keep the full execution history

H_{t}

in the prompt rather than just the latest error?

So the agent won't re-try a fix it already saw fail; the history encodes "what's been tried and what happened," preventing it from cycling through the same dead ends.

Why generate tests before the implementation in TDD?

Tests turn vague natural-language requirements into concrete, checkable assertions, exposing edge cases up front and giving the agent an automatic way to verify its own later code.

Why check divisors only up to

n

when testing primality?

Divisors come in pairs

a \times b = n

; one of each pair is always

\leq n

, so if none below

n

divides

n

, none above it can either — checking the larger half is redundant.

Why is the naive "add defensive checks everywhere" a trap in error recovery?

It pattern-matches on error keywords instead of reasoning about data flow, so it silences the message while leaving the true cause (bad loop bounds, uninitialized counters) intact — the bug returns elsewhere.

Why blend keyword (BM25) and embedding retrieval with a balance parameter

α

To tune how much weight goes to exact-name matches versus meaning-based matches; small

α

leans on precise identifiers, large

α

leans on conceptual synonyms.

Why does the success-probability formula reward independence between attempts?

If attempts are independent, each failure is a separate coin flip, so failing all

k

(1 - p)^{k}

, which shrinks fast; correlated attempts share the same blind spot and don't multiply down like that.

Edge cases

What does

1 - (1 - p)^{k}

give when

p = 0

(the agent can never solve it)?

Zero for every

k

—

(1 - 0)^{k} = 1

, so no number of independent tries helps. Iteration cannot rescue a task the model is fundamentally unable to do.

What does the same formula give when

p = 1

1 - (1 - 1)^{k} = 1

for any

k \geq 1

— a guaranteed solver succeeds on the first try, and extra iterations add nothing.

What does pass@k give at

k = 0

(you submit nothing)?

Exactly

0

: by convention

(0 n - c) = (0 n) = 1

, so

1 - \frac{1}{1} = 0

. Drawing zero samples can never yield a correct one, which is why the metric is only meaningful for

k \geq 1

What is pass@k when

k = n

(you keep every sample you drew)?

If there is at least one correct sample (

c \geq 1

) it is

1

, since

(k n - c) = (n n - c) = 0

; the whole point of drawing everything is you never miss an existing correct answer.

What is pass@k when the pool has zero correct samples (

c = 0

Exactly

0

, because

(k n - c) = (k n)

and the fraction equals 1, so

1 - 1 = 0

. You cannot select a correct answer that was never generated.

What does pass@k mean when

k > n

— you ask for more attempts than samples exist?

It is not defined by the standard formula, because

(k n) = 0

when

k > n

(you cannot choose

k

items from fewer than

k

), giving a divide-by-zero. In practice you either cap

k

n

or must generate at least

k

samples first; asking for more draws than you have is meaningless.

How should the agent behave when execution produces no error and no output (a silent, empty run)?

Treat empty output as a signal too, not success — often it means the code ran but printed nothing or the test path wasn't reached. The agent should add a print/assert to confirm the logic actually executed before declaring completion.

What happens at the very first step

t = 0

when the history

H_{0}

is empty?

The context contains only

R

(requirements) and system instructions with no past attempts, so the first action is a pure one-shot generation — the loop's feedback only begins once the first execution fills

H_{1}

Recall Two-line self-test

The agent loop's defining feature over plain completion? ::: A feedback loop with the environment — run, read output/error, edit, repeat. pass@k with $c = 0$ correct samples? ::: Exactly 0 — you cannot pick a correct sample that was never generated.