Evaluation of LLMs (benchmarks, LLM-as-judge): Worked examples

This page is a firing range. The parent note gave you two engines — Pass@k for objective benchmarks and Cohen's Kappa for LLM-as-judge agreement. Here we fire them at every case that can appear: the easy ones, the sign traps, the degenerate zero-inputs, the limiting values, and the exam twists.

The scenario matrix

Before defining a single symbol again, here is the map of every kind of situation the two formulas face. Read the columns as "what makes this case special."

Cell	Formula	The special condition	What breaks / what to watch
C1	Pass@k	Typical case: $0 < c < n$	Nothing special — baseline
C2	Pass@k	Degenerate low: $c = 0$ (no sample ever passed)	Estimator says $0$ ; must not error
C3	Pass@k	Degenerate high: $c = n$ (every sample passes)	Estimator says $1$ ; combinatorial form has $(k 0)$
C4	Pass@k	Limiting $k \to$ large vs $k = 1$	How the score grows with more tries
C5	Pass@k	Unbiased combinatorial estimator on ONE problem	Sampling-without-replacement form
C6	Kappa	Typical moderate agreement	Baseline judge-vs-human
C7	Kappa	Degenerate: judge and human always pick the same side ( $p_{e} \to 1$ )	Denominator $1 - p_{e} \to 0$
C8	Kappa	Negative kappa: judge does worse than chance	Sign of $κ$ goes below $0$
C9	Both	Real-world word problem: end-to-end eval report	Combine both metrics
C10	Kappa	Exam twist: given $κ$ and $p_{e}$ , recover $p_{o}$	Algebra inversion

We now hit every cell.

Recall the two engines (symbols re-earned)

Recall Pass@k — what each letter means

$n$ ::: total number of code samples drawn for one problem $c$ ::: how many of those $n$ samples passed all hidden tests $k$ ::: how many samples we "get to try" (report at) $Pass@ k$ ::: the probability that at least one of $k$ tries succeeds

Recall Cohen's Kappa — what each letter means

$p_{o}$ ::: observed agreement, the fraction of comparisons where judge and human chose the same winner $p_{e}$ ::: agreement expected by pure chance $κ$ ::: agreement beyond chance, on a scale where $1$ is perfect and $0$ is "no better than luck"

The two forms we will use:

$Pass@ k = 1 - \frac{( k n - c )}{( k n )}, κ = \frac{p _{o} - p _{e}}{1 - p _{e}}$

The symbol $(k n)$ (" $n$ choose $k$ ") counts how many ways to pick $k$ items out of $n$ when order does not matter. We use it because "which $k$ samples did the judge happen to draw" is exactly an unordered choice.

C1 — Pass@k, the typical case

Worked example C1:

$0 < c < n$ A problem has $n = 10$ samples generated. Of them, $c = 3$ passed all tests. Report Pass@2.

Forecast: Guess before computing — should Pass@2 be bigger or smaller than the single-sample rate $3/10 = 0.30$ ? (Two tries should help, so bigger.)

Count the failing samples: $n - c = 10 - 3 = 7$ . Why this step? Pass@k is easiest as "1 minus the chance both tries land in the failing pile."
Chance both drawn samples are failures (drawing 2 of the 7 failures out of all 10): $\frac{( 2 7 )}{( 2 10 )} = \frac{21}{45} .$ Why this step? $(2 7)$ = ways to pick 2 losers; $(2 10)$ = ways to pick any 2. Their ratio is the probability of "all failed."
Subtract from 1: $Pass@ 2 = 1 - \frac{21}{45} = \frac{24}{45} \approx 0.5333.$

Verify: $0.533 > 0.30$ ✓ (two tries beat one). And it is below $1$ because success is not guaranteed. Units: a probability, dimensionless, in $[0, 1]$ ✓.

C2 — Pass@k degenerate low ( $c = 0$ )

C3 — Pass@k degenerate high ( $c = n$ )

C4 — Pass@k limiting behaviour in $k$

The next figure shows why more tries always help but with shrinking returns.

Figure — Evaluation of LLMs (benchmarks, LLM-as-judge)

C5 — Pass@k unbiased combinatorial estimator, single problem

C6 — Kappa, typical moderate case

This reproduces the parent's headline number, so you can trust the machinery.

C7 — Kappa degenerate ( $p_{e} \to 1$ )

C8 — Kappa negative (worse than chance)

C9 — Real-world word problem (both engines)

Worked example C9: an end-to-end evaluation report

You are comparing two chat models. For HumanEval, model X solved at least one of $k = 5$ samples on $c = 82$ of $n = 164$ problems — report Pass@5 as the simple estimator $1 - (1 - c / n)^{5}$ ... wait, that misuses the formula. The parent's simple estimator uses the per-problem pass rate. Here instead we're given that across 164 problems, 82 had "at least one success." That is already Pass@5 directly: $Pass@ 5 = \frac{82}{164} = 0.50.$ Then, on AlpacaEval, an LLM judge and humans agree on 170 of 200 pairwise comparisons; humans pick X 45% of the time, judge picks X 48% of the time. Report $κ$ .

Forecast: Pass@5 = 0.50 exactly; kappa should be solidly "substantial" since $p_{o} = 0.85$ is well above chance.

$Pass@ 5 = 82/164 = 0.50$ . Why this step? "Fraction of problems with $\geq 1$ passing sample" is the definition of Pass@k when we count per problem.
$p_{o} = 170/200 = 0.85$ .
$p_{e} = 0.45 \times 0.48 + 0.55 \times 0.52 = 0.216 + 0.286 = 0.502.$
$κ = \frac{0.85 - 0.502}{1 - 0.502} = \frac{0.348}{0.498} \approx 0.6988.$

Verify: $κ \approx 0.70$ sits in $[0.6, 0.8]$ = substantial ✓. Report reads: "Model X passes half of HumanEval@5; the judge substantially agrees with humans, so its AlpacaEval win rates are trustworthy." Cross-links: this pipeline feeds RLHF reward signals and interacts with the Alignment Tax.

C10 — Exam twist: invert the formula

Recall Quick self-test

Pass@k when $c = 0$ ::: always $0$ — no try can succeed Pass@k when $c = n$ ::: always $1$ — any try succeeds; note $(k 0) = 0$ Why report $p_{e}$ next to $κ$ ::: when one label dominates, $p_{e} \to 1$ and $κ$ misleads even at high $p_{o}$ Sign of $κ$ below zero means ::: judge agrees worse than chance — likely a systematic bias Invert kappa: $p_{o} =$ ::: $κ (1 - p_{e}) + p_{e}$

Related tooling and ideas: Prompt Engineering Best Practices shape the judge prompt, Few-Shot Learning can calibrate the judge, Constitutional AI supplies judging criteria, and evaluation of RAG systems layers retrieval-quality on top of these same metrics. The judge's verdict tokens are A, B, or Tie.