Evaluation of LLMs (benchmarks, LLM-as-judge): Exercises

Before we start, one shared picture of the whole evaluation world:

Figure — Evaluation of LLMs (benchmarks, LLM-as-judge)

The green road (benchmarks) is used when a ground-truth answer exists. The coral road (LLM-as-judge) is used when quality is subjective. Every exercise below sits on one of these two roads.

Level 1 — Recognition

Goal: can you name the right tool and read the notation?

Exercise 1.1

Match each task to whether it should be scored by a benchmark (objective ground truth) or by an LLM-as-judge (subjective quality):

"Solve: $17 \times 23$ ."
"Write a comforting message to someone who lost a pet."
"What is the capital of France? (A) Berlin (B) Paris (C) Rome (D) Madrid"
"Which of these two summaries is more helpful?"

Recall Solution 1.1

A benchmark needs a single checkable answer; LLM-as-judge handles open-ended quality.

Benchmark — one correct number ( $391$ ), exact-match scoring.
LLM-as-judge — no single right message; "comforting" is subjective.
Benchmark — multiple-choice with one correct letter (B), accuracy scoring.
LLM-as-judge — "more helpful" is a preference, no ground truth.

Rule of thumb: if you can write the answer key in advance, use a benchmark.

Exercise 1.2

In the pairwise judge prompt, the judge is told to output one of [[A]], [[B]], or [[Tie]]. In plain words, what does each verdict mean, and what does the judge compare?

Recall Solution 1.2

The judge reads one instruction and two responses, then decides which response better satisfies criteria like accuracy/helpfulness/safety.

A — response A is better.
B — response B is better.
Tie — neither is clearly better.

Crucially, the judge is comparing the two responses to each other, not to a hidden gold answer. That is exactly why we use a judge here: there is no gold answer to compare against.

Level 2 — Application

Goal: plug numbers into the formulas built in the parent note.

Exercise 2.1 (Accuracy)

A model answers $14, 042$ MMLU questions and gets $11, 964$ correct. What is its accuracy, as a percentage rounded to one decimal place?

Recall Solution 2.1

What accuracy is: the fraction of items the model got exactly right. $Accuracy = \frac{correct}{total} = \frac{11 , 964}{14 , 042}$ Why division: each question contributes equally, so we average a 0/1 score over all questions. $= 0.85202 \dots \approx 85.2%$

Exercise 2.2 (Simple Pass@k)

A coding benchmark has $n = 200$ problems. For $k = 5$ samples each, a model produces at least one passing sample on $c = 150$ of them. Using the simple estimator $Pass@ k = 1 - (1 - \frac{c}{n})^{k},$ compute Pass@5. (Round to 3 decimals.)

Recall Solution 2.2

What the formula says: the chance of getting at least one correct in $k$ tries equals $1$ minus the chance all $k$ fail.

Here $\frac{c}{n} = \frac{150}{200} = 0.75$ , so the single-try failure rate is $1 - 0.75 = 0.25$ . $Pass@ 5 = 1 - (0.25)^{5} = 1 - 0.0009766 = 0.99902 \approx 0.999$ Why so high? With a 75% per-try success rate, failing five times in a row is astronomically unlikely — the exponent crushes it.

Exercise 2.3 (Observed agreement)

An LLM judge and a human agree on the winner in $80$ out of $100$ pairwise comparisons. What is the observed agreement $p_{o}$ ?

Recall Solution 2.3

$p_{o}$ is just the raw agreement fraction: $p_{o} = \frac{a}{n} = \frac{80}{100} = 0.80$ Warning for later: this number looks great, but it ignores agreement that could happen by pure luck. Level 3 fixes that.

Level 3 — Analysis

Goal: use the pieces together and interpret the result.

Exercise 3.1 (Cohen's Kappa)

Reuse Exercise 2.3: $p_{o} = 0.80$ over $100$ comparisons. Suppose the human picks response A $60%$ of the time and the judge picks A $55%$ of the time. Compute the chance-corrected agreement $κ$ , and classify it.

Recall Solution 3.1

Step 1 — expected-by-chance agreement $p_{e}$ . If both parties chose independently, they'd still coincide sometimes. They agree on "A" with probability $p_{A} \cdot q_{A}$ and on "B" with probability $p_{B} \cdot q_{B}$ : $p_{e} = (0.60) (0.55) + (0.40) (0.45) = 0.33 + 0.18 = 0.51$ Why the products: independent events → multiply; two ways to accidentally agree (both A, both B) → add.

Step 2 — subtract out the luck. $κ = \frac{p _{o} - p _{e}}{1 - p _{e}} = \frac{0.80 - 0.51}{1 - 0.51} = \frac{0.29}{0.49} = 0.5918 \dots \approx 0.59$ Reading the scale: $0.4 - 0.6$ = moderate agreement. So despite the flashy $80%$ raw number, once we remove the $51%$ we'd expect from chance, the judge is only moderately aligned with the human.

Exercise 3.2 (Why Kappa < Accuracy)

In 3.1, raw agreement was $0.80$ but $κ$ was only $0.59$ . Explain in one or two sentences why $κ$ dropped, and describe a scenario where $κ$ would be negative.

Recall Solution 3.2

$κ$ dropped because a binary A/B choice already collides ~ $51%$ of the time by luck; $κ$ measures agreement above that floor, and rescales the remaining $1 - p_{e} = 0.49$ of "room to agree" to a max of $1$ .

$κ < 0$ happens when $p_{o} < p_{e}$ — the judge and human agree less often than random chance would predict, i.e. they are systematically anti-correlated (when the human likes A, the judge tends to pick B). That is worse than a coin flip.

Level 4 — Synthesis

Goal: combine formulas and reasoning to make a decision.

Exercise 4.1 (Unbiased Pass@k vs simple estimator)

The parent note gives the unbiased Pass@k estimator for a single problem with $n$ total samples and $c$ passing ones: $Pass@ k = 1 - \frac{( k n - c )}{( k n )} .$ For one problem, we drew $n = 10$ samples, of which $c = 3$ passed. Compute the unbiased Pass@3. Then compute the naive version $1 - (1 - \frac{c}{n})^{3}$ and explain which is correct and why.

Recall Solution 4.1

Unbiased (sampling without replacement). The chance all $3$ chosen samples come from the $n - c = 7$ failing ones is $\frac{( 3 7 )}{( 3 10 )} = \frac{35}{120} = 0.29167.$ So $Pass@ 3 = 1 - 0.29167 = 0.70833 \approx 0.708.$ Naive (sampling with replacement). $1 - (1 - \frac{3}{10})^{3} = 1 - (0.7)^{3} = 1 - 0.343 = 0.657.$ Which is right? We already generated a fixed pool of $10$ samples and are asking "if I pick $3$ of these, does one pass?" — that is drawing without replacement, so the unbiased combinatorial form ( $0.708$ ) is correct. The naive form assumes we re-roll the true probability each time (with replacement) and here underestimates Pass@3. Why the gap: without replacement, each failure you draw makes the remaining pool richer in successes, nudging Pass@k up relative to the independent-trials assumption.

Exercise 4.2 (Position bias defense)

An LLM judge is run on $100$ instruction pairs. When the better answer is placed as A, the judge picks it $92$ times. When the same better answer is placed as B, the judge picks it only $74$ times. (a) What phenomenon does this reveal? (b) Propose a concrete fix and state what the "true" win rate estimate becomes after applying it.

Recall Solution 4.2

(a) Position bias. The judge favors whichever response sits in slot A regardless of content — evidence: the same answer scores $92%$ in slot A but $74%$ in slot B.

(b) Fix — swap-and-average (order randomization). Run every pair twice, once as (A,B) and once swapped to (B,A), and average. The debiased estimate of how often the truly-better answer wins is $\frac{92 + 74}{2 \times 100} = \frac{166}{200} = 0.83 = 83%.$ This cancels the constant slot-A advantage. (A stricter variant: only count a "win" when the better answer wins in both orders, discarding order-dependent flips as ties.)

Level 5 — Mastery

Goal: end-to-end judgement call combining every concept and the wider chapter.

Exercise 5.1 (Choosing an evaluation stack)

You fine-tuned a chatbot with RLHF and want to prove it improved. You have three claims to evaluate:

(i) "It got better at grade-school math."
(ii) "It's more helpful in open-ended chat."
(iii) "Our automated judge actually reflects human taste."

For each claim, name the evaluation method, the metric, and the one number that would make you distrust the result.

Recall Solution 5.1

Claim	Method	Metric	Red flag
(i) math	Benchmark (e.g. GSM8K)	Exact-match accuracy	A big jump that vanishes on held-out/unseen problems → benchmark contamination (leakage into training data)
(ii) chat	LLM-as-judge, pairwise vs old model	Win rate (both orders, swap-averaged)	Win rate near $50%$ , or a rate that flips when you swap A/B → position bias not controlled
(iii) judge trust	Compare judge to human labels	Cohen's $κ$	$κ < 0.4$ → judge disagrees with humans beyond chance; a high raw accuracy alone is not enough

Key mastery point: (i) needs ground truth, (ii) needs a judge, and (iii) needs a chance-corrected agreement to validate the judge itself — you cannot trust the win rate in (ii) until $κ$ in (iii) is at least substantial ( $\geq 0.6$ ).

Exercise 5.2 (The full alignment-tax tradeoff)

Your RLHF model wins $65%$ of open-ended chat comparisons (judged, swap-averaged) but its GSM8K accuracy dropped from $58%$ to $52%$ . (a) Name this tradeoff. (b) Compute the relative drop in math accuracy. (c) Given both numbers, argue whether to ship — and name one other chapter technique you'd try to recover the loss.

Recall Solution 5.2

(a) This is the Alignment Tax: making a model more helpful/aligned can cost raw capability on objective tasks.

(b) Relative drop in math accuracy: $\frac{58 - 52}{58} = \frac{6}{58} = 0.10345 \approx 10.3%.$ (Absolute drop is $6$ points; relative drop is about $10.3%$ of the original score.)

(c) Decision: the chat win rate of $65%$ is a clear, judge-verified helpfulness gain, while the math cost is a modest $\sim 10%$ relative drop. If chat is the product's primary use, ship — but mitigate the tax. One recovery lever from this chapter: Constitutional AI (align with model-generated principles to reduce reliance on capability-eroding human labels), or preserve math skill with Few-Shot Learning prompting (in-context math exemplars at inference time) so you don't need to re-train the weights. Either way, re-run both the benchmark and the judge after the fix — never trust one road alone.

Recall One-screen summary

Objective task ::: benchmark + accuracy / Pass@k / exact-match Subjective task ::: LLM-as-judge, pairwise win rate Pass@k meaning ::: probability at least one of k samples passes Why $1 - (\cdot)^{k}$ ::: subtract the chance that all k samples fail Why unbiased Pass@k uses binomials ::: samples are drawn without replacement from a fixed pool Cohen's $κ$ purpose ::: agreement corrected for chance ( $p_{o}$ minus $p_{e}$ , rescaled) $κ < 0$ means ::: judge and human agree less than random chance Position bias fix ::: evaluate both A/B orders and average Alignment tax ::: capability lost while gaining alignment