4.4.16 · D5Alignment, Prompting & RAG
Question bank — Evaluation of LLMs (benchmarks, LLM-as-judge)
Before we start, one shared vocabulary reminder built from zero:
True or false — justify
The statement is either true or false — but the reveal explains why, which is the actual test.
A high MMLU score proves a model can reason deeply.
False — MMLU is multiple-choice knowledge recall; a model can pattern-match the right letter without any reasoning chain, which is why the parent lists "measures memorization, not reasoning depth" as its limitation.
Pass@1 and Pass@10 measure the same thing at different sample counts.
True in spirit — both estimate — but they answer different questions: Pass@1 is "solves on the first try", Pass@10 is "solves if allowed 10 attempts", so Pass@10 is always Pass@1.
If a judge and a human agree 80% of the time, the judge is 80% reliable.
False — raw agreement ignores luck; in a binary choice ~50% agreement happens by chance, so you must chance-correct with Cohen's Kappa before calling it reliable.
A benchmark with ground truth never needs an LLM judge.
Mostly true — if an exact answer key exists (MMLU, GSM8K), exact-match or accuracy suffices; but partially-correct free-form answers can still need a judge to award partial credit.
Two models scoring identically on MMLU are equally good assistants.
False — MMLU says nothing about helpfulness, safety, tone, or instruction-following; a model can ace trivia yet be useless or unsafe in conversation (Alignment Tax shows these can even trade off).
Cohen's Kappa of 0 means the judge is useless.
True in the sense that means the judge agrees with humans no more than random guessing would — all its "agreement" was chance.
A negative Kappa is impossible.
False — occurs when observed agreement is below chance, meaning the judge systematically disagrees with humans (worse than a coin flip).
Higher win-rate against a baseline always means a better model.
False — win-rate depends entirely on which baseline you chose; beating a weak baseline 90% of the time proves little, so the baseline must be stated.
Spot the error
Each line contains a flawed claim or setup. Name the flaw.
"We estimated as then computed Pass@k — perfectly unbiased."
The plug-in is biased; the parent's unbiased estimator uses combinations to correct for sampling without replacement.
"Our judge is GPT-4 and one of the two answers it compares is also from GPT-4 — no problem."
Self-preference bias: judges tend to favour outputs from their own model family, inflating that model's win-rate; the setup is contaminated.
"We put Response A first and B second every time to keep things consistent."
Position bias — judges often favour whichever answer appears first (or second) regardless of quality; you must randomize order and ideally average both orderings.
" is the expected chance agreement."
Incomplete — it omits the B term; expected agreement is , summing over both possible matching choices.
"The model scored 95% on GSM8K, so it truly reasons through multi-step math."
Could be contamination — if GSM8K problems leaked into training, the score reflects memorization; you'd verify on a held-out or newly written test set.
"Pass@k went up when we raised sampling temperature, so the model got smarter."
The model didn't change; higher temperature just increases sample diversity, so at least one of samples is more likely correct — it inflates Pass@k without improving per-sample skill .
"Judge gave Accuracy 8, Helpfulness 9, Safety 10, so overall quality is 9."
Averaging assumes the three axes are comparable and equally weighted; a Safety of 10 cannot compensate for a factual error, so a single blended number can hide a disqualifying flaw.
Why questions
Why compute Pass@k via the complement instead of directly?
The event "at least one of succeeds" is a messy union of overlapping cases, while "all fail" is one clean product under independence — the complement is far easier to compute.
Why subtract chance agreement in Kappa instead of just reporting raw agreement?
Because a binary judge scores ~50% by luck alone; subtracting measures skill above random, so a hard task's genuine agreement isn't flattered by easy coin-flip matches.
Why use pairwise A-vs-B comparison instead of absolute 1–10 scores?
Humans (and judges) are far more consistent at relative judgments ("B is better") than at calibrating an absolute number, so pairwise preferences give more stable, reproducible rankings.
Why is an LLM judge preferred over hiring human raters at scale?
Human rating costs $0.50–5 each and takes days with only 60–80% inter-annotator agreement; a judge is cheap, instant, and reproducible — provided its Kappa against humans is validated first.
Why does the judge in the summarization example prefer the detailed answer B?
It was trained on human preferences that reward informativeness, so it catches that B supplies a mechanism, a number (C) and consequences while A is vague — mirroring how RLHF instilled those preferences.
Why can two valid explanations of "why the sky is blue" both be scored highly, breaking exact-match?
There is no single ground-truth string; exact-match rewards only one phrasing, so open-ended tasks need a judge assessing meaning, not surface tokens.
Why might a model trained with Constitutional AI score lower on a raw helpfulness benchmark?
Safety guardrails make it refuse or hedge on borderline requests, which a pure-helpfulness judge may penalize — the classic Alignment Tax trade-off between being maximally helpful and being safe.
Edge cases
What is Pass@k when (no sample ever passed)?
; the model solved nothing, correctly giving zero — and the unbiased form gives failures, also .
What is Pass@k when (every problem solved)?
; a perfect 100%, as expected — and unbiased (can't choose from zero remaining failures) gives .
What does Pass@1 equal in terms of the estimated per-sample success ?
Exactly itself, since — Pass@1 is the plain success rate.
What is Kappa when (observed equals chance)?
— the judge added nothing beyond luck, regardless of how high looks.
What is Kappa when (perfect agreement)?
, the maximum, no matter what chance level was.
What happens to Kappa's denominator as ?
, so Kappa becomes numerically unstable — on a task where humans almost always pick the same side, the metric is dominated by tiny fluctuations and should be reported cautiously.
What should you report when a task has ground truth for some items and none for others?
A hybrid — exact-match/accuracy on the objective portion and judge-based (chance-corrected) scores on the subjective portion, kept as separate numbers rather than one misleading average.
What is the correct Pass@k reading if but the model gave 5 samples in the log?
You must select exactly one sample per problem to estimate Pass@1; using all 5 would silently measure Pass@5 — the sample count in the formula must match how the score is defined.
Recall Two-line self-test
Why does Pass@k use the complement rule? ::: Because "all fail" is a clean product while "at least one succeeds" is a messy union. Why chance-correct agreement into Kappa? ::: Because ~50% agreement is free in a binary choice, so only agreement above reflects real judging skill.