Evaluation of LLMs (benchmarks, LLM-as-judge): Visual walkthrough

Everything below rests on one idea from the parent note: a benchmark runs the code and checks if it passes hidden tests. So each attempt is either a ✅ (passes all tests) or a ❌ (fails at least one). Nothing in between. Hold onto that.

Step 1 — One attempt is a biased coin

WHAT. Picture the model making a single attempt at one problem. The outcome is binary: ✅ pass or ❌ fail. We invent a number $p$ for "how often the ✅ side comes up."

WHY a probability at all? Because the model is random — same prompt, different sampled words each time (temperature > 0). We cannot predict one attempt, but we can talk about how often ✅ shows up. That "how often" is exactly what $p$ names.

PICTURE. A coin painted green on one face (✅, area $p$ ) and red on the other (❌, area $1 - p$ ). The green fraction is $p$ .

Figure — Evaluation of LLMs (benchmarks, LLM-as-judge)

Step 2 — Failing all $k$ tries is the easy thing to count

WHAT. We want $P (at least one ✅ in k)$ . Counting "at least one" is annoying — it could be 1, or 2, ... up to $k$ successes, and we'd have to add all those cases. So we flip the question and count the one way things go completely wrong: every attempt is ❌.

WHY subtraction is legal. For any event $E$ , $P (E) = 1 - P (not E) .$ Here $E$ = " $\geq 1$ success", so "not $E$ " = "all $k$ failed". Term by term:

$1$ — total probability of all outcomes (something must happen).
$P (all k failed)$ — the single slice we subtract off.

PICTURE. The whole probability bar is length $1$ . We shade the small "all-fail" chunk in red; whatever's left (blue) is our answer.

Step 3 — Independent tries multiply

WHAT. Each attempt fails with probability $1 - p$ (the red face). If the tries don't influence each other, the chance that attempt 1 fails and 2 fails and ... and $k$ fails is the product of $k$ copies of $(1 - p)$ .

WHY multiply and not add? "AND across independent events" is multiplication — think of a tree. Each try branches into ✅ or ❌; walking the all-red path means multiplying the red probability at every branch: $P (all fail) = try 1 (1 - p) \times try 2 (1 - p) \times \dots = (1 - p)^{k} .$

$(1 - p)$ — probability one attempt lands ❌.
exponent $k$ — how many attempts, each an independent ❌.

PICTURE. A branching tree, $k$ levels deep. The single all-red path down the left is highlighted; its length shrinks as $k$ grows.

Plugging Step 3 into Step 2: $Pass@ k = 1 - (1 - p)^{k}$

$(1 - p)^{k}$ — everything went wrong.
$1 - (\dots)$ — so something went right.

Step 4 — We don't know $p$ , so we estimate it

WHAT. The formula needs $p$ , but $p$ is the true pass rate we can never see. All we have is data: over $n$ real problems, we watched how often the model succeeded. We replace $p$ by the observed success fraction $\overset{p}{^}$ .

WHY the raw fraction? It is the maximum-likelihood estimate: among all values of $p$ , the observed count $c$ was most probable precisely when $p = c / n$ . In plain words — the guess that makes what you actually saw the least surprising. If 30 of 100 tries passed, betting $p = 0.30$ is the least-surprised bet.

PICTURE. A jar of 100 marbles, 30 green; you scoop them and estimate "green-ness" $= 30/100$ .

Now the usable formula from the parent note appears: $Pass@ k = 1 - (1 - \frac{c}{n})^{k} .$

Step 5 — Every case: what Pass@k does at the extremes

WHAT. A formula you trust is one you've watched at its boundaries. Read $Pass@ k = 1 - (1 - p)^{k}$ across all inputs.

Input	Meaning	Pass@k
$p = 0$	model never solves it	$1 - (1 - 0)^{k} = 1 - 1 = 0$
$p = 1$	model always solves it	$1 - (1 - 1)^{k} = 1 - 0 = 1$
$k = 1$	one attempt only	$1 - (1 - p)^{1} = p$
$k \to \infty$	infinite retries, any $p > 0$	$1 - (1 - p)^{k} \to 1$

WHY each makes sense.

$p = 0$ : no green face exists → all tries fail no matter how many → $0$ . ✅
$p = 1$ : first try already wins → $1$ . ✅
$k = 1$ : Pass@k is just $p$ ; the whole machinery reduces to one coin flip. ✅
$k \to \infty$ : as long as some green exists ( $p > 0$ ), keep flipping and you're guaranteed to hit it eventually → $1$ . This is why bigger $k$ always flatters a model. ✅

PICTURE. Curves of Pass@k versus $p$ , one curve per $k$ . All pass through $(0, 0)$ and $(1, 1)$ ; higher $k$ bows harder toward the top-left.

Step 6 — The honest estimator (sampling without replacement)

WHAT. In real HumanEval you don't flip infinitely. You draw a fixed pool of $N$ samples per problem, count how many pass ( $C$ ), then ask: if I'd picked $k$ of these $N$ , what's the chance all $k$ were failures? Since you're picking from a finite pool without putting them back, the naive $(1 - c / n)^{k}$ is slightly biased. The parent note's boxed formula fixes this with counting, not multiplication.

WHY combinations here instead of a power? When you remove a failed sample and can't draw it again, the odds shift each pick — so you can't just multiply one fixed $(1 - p)$ . Counting arrangements handles the shifting odds exactly. (If the pool were infinite, $(k N - C) / (k N) \to (1 - C / N)^{k}$ and Step 3's formula returns.)

PICTURE. A row of $N$ cards, $C$ green and $N - C$ red. We fan out one hand of $k$ cards drawn without replacement; the "all-red hand" is the failure event we're counting.

The one-picture summary

The whole story on one canvas: a coin ( $p$ ) → the all-fail red slice ( $(1 - p)^{k}$ ) → subtract from $1$ → estimate $p$ from data → the rising family of curves → the finite-pool correction.

Recall Feynman retelling — say it like you'd explain to a friend

The model is a slightly unreliable machine. On any single try it succeeds a fraction $p$ of the time — that's our green coin. We give it $k$ shots and only care whether any shot lands. Counting "any" is hard, so we count the opposite: the one way it's a total flop is every shot missing, which (if shots don't affect each other) is $(1 - p)$ multiplied by itself $k$ times, i.e. $(1 - p)^{k}$ . Whatever's left over, $1 - (1 - p)^{k}$ , is the chance of at least one win — that's Pass@k. We never actually know $p$ , so we just use the fraction of problems it solved, $c / n$ . Sanity checks: a hopeless model ( $p = 0$ ) scores $0$ no matter how many tries; a perfect one scores $1$ ; one try makes the whole thing collapse to plain $p$ ; and infinite tries push any nonzero model to $100%$ — which is exactly why you must compare models at the same $k$ . Finally, since a real benchmark draws from a finite bag of samples you can't redraw, we swap the "multiply" for a "count the hands" formula with combinations, and that removes the small bias.

Recall

What does $p$ mean in one sentence? ::: The true fraction of single attempts that pass all tests for a problem. Why do we compute $1 - (1 - p)^{k}$ instead of adding up success cases? ::: "At least one success" and "all fail" are complementary, so it's easier to compute the single all-fail case and subtract from 1. Why is $(1 - p)$ raised to the power $k$ ? ::: The $k$ attempts are independent, and "all fail" means every one fails — independent ANDs multiply. What is $\overset{p}{^} = c / n$ and why that value? ::: The maximum-likelihood estimate of the pass rate — the guess that makes the observed data least surprising. What happens to Pass@k as $k \to \infty$ for any $p > 0$ ? ::: It approaches 1 — with enough retries any nonzero-chance model eventually succeeds. Why does HumanEval use the combinatorial $(k N - C) / (k N)$ form? ::: Samples are drawn from a finite pool without replacement, so a fixed- $p$ power is biased; counting arrangements is exact.