Evaluation of LLMs (benchmarks, LLM-as-judge)

Core Problem: The Evaluation Gap

Language model evaluation faces three challenges:

1. Capability diversity: LLMs do everything from math to creative writing to code
2. Output subjectivity: "Good" varies by task (factual vs. creative vs. helpful)
3. Human evaluation cost: Expert review doesn't scale to millions of outputs

Solution approach: Combine objective benchmarks (for measurable skills) with model-based judges (for subjective quality).

Part 1: Benchmark Evaluation

How Benchmarks Work

The pipeline:

Model → Prompt with task → Generate output → Compare to answer → Compute metric

Key benchmark types:

| Type | Measures | Example Metric | |------|----------|------|-----| | Multiple choice | Knowledge recall | MMLU (57 subjects) | Accuracy | | Code generation | Programming ability | HumanEval, MBPP | Pass@k (% passing tests) | | Math word problems | Reasoning | GSM8K, MATH Exact match | | Reading comprehension | Understanding | SQuAD, DROP | F1 score | | Instruction following | Alignment | MT-Bench, AlpacaEval | Win rate vs baseline |

Deriving Pass@k from First Principles

Problem: Model generates $k$ code samples per problem. What's the probability ≥1 is correct?

Setup:

• $n$ = total problems
• $c$ = problems where≥1 of $k$ samples passed all tests
• $p$ = true probability a single sample is correct (unknown)

Derivation: $\text{Pass@}k = \mathbb{P}(\text{≥1 correct in } k \text{ samples})$

Why subtract? Easier to compute complement: $= 1 - \mathbb{P}(\text{all } k \text{ wrong})$

If samples are independent: $= 1 - (1-p)^k$

But we don't know $p$! Estimate from data: $\hat{p} = \frac{c}{n}$

Why this estimator? Maximum likelihood: observed success rate = best guess of true rate.

Final formula: $\boxed{\text{Pass@}k = 1 - \left(1 - \frac{c}{n}\right)^k}$

Part 2: LLM-as-Judge

Why We Need It

Scenarios where benchmarks fail:

• Creative tasks: "Write a haiku about loneliness" (no ground truth)
• Open-ended QA: "Explain why the sky is blue" (many valid explanations)
• Conversational ability: Multi-turn dialogues (context-dependent quality)
• Alignment: Is the response helpful and safe? (subjective judgment)

Human evaluation problems:

• Expensive: $0.50–5 per rating, millions needed for large-scale tuning
• Slow: Days to weeks for crowdworker consensus
• Inconsistent: Inter-annotator agreement often 60–80%

Solution: Train/prompt a strong LM to mic human preferences.

How LM-as-Judge Works

Pairwise comparison setup (most common):

Prompt to judge:
---
Instruction: {user_question}

Response A: {model_1_output}
Response B: {model_2_output}

Which response is better? Consider accuracy, helpfulness, safety.
Choose: [[A]] or [[B]] or [[Tie]]
---

Judge outputs: "A"
Aggregate over many examples: Model 1 wins 65% vs Model 2 → 1 ranked higher.

Absolute scoring setup:

Rate this response on a scale of 1-10 for:
- Accuracy (factually correct?)
- Helpfulness (answers the question?)
- Safety (avoids harmful content?)

Response: {output}

Judge outputs: "Accuracy: 8, Helpfulness: 9, Safety: 10"

Deriving Agreement Metrics from First Principles

Problem: How well does the LM judge agree with humans?

Setup:

• $n$ = total comparisons
• $a$ = agreements (judge and human pick same winner)
• $d$ = disagreements

Cohen's Kappa (accounts for chance agreement):

$\kappa = \frac{p_o - p_e}{1 - p_e}$

Where:

• $p_o$ = observed agreement = $\frac{a}{n}$
• $p_e$ = expected agreement by chance

Why subtract chance? Random guessing gets ~50% agreement in binary choice. We want agreement beyond luck.

Deriving $p_e$: If human picks A with probability $p_A$ and judge picks A with probability $q_A$: $p_e = p_A \cdot q_A + p_B \cdot q_B$

Why product? Probability both pick A by chance = (prob human picks A) × (prob judge picks A), assuming independence.

Example calculation:

• 100 comparisons, 80 agreements → $p_o = 0.80$
• Human picks A 60% of time, judge picks A 55% of time
• $p_e = 0.60 \times 0.55 + 0.40 \times 0.45 = 0.33 + 0.18 = 0.51$
• $\kappa = \frac{0.80 - 0.51}{1 - 0.51} = \frac{0.29}{0.49} = 0.59$ (moderate agreement)

Part 3: Key Benchmarks in Practice

MMLU (Massive Multitask Language Understanding)

• What: 57 subjects (elementary math to professional law), 14k multiple choice questions
• Why it matters: Broad knowledge test, correlates with "feels smart" to users
• Limitation: Measures memorization, not reasoning depth

GSM8K (Grade School Math 8K)

• What: 8,500 grade-school word problems requiring multi-step reasoning
• Why it matters: Tests chain-of-thought, not just formula recall
• Example: "If a train travels 60 mph for 2.5 hours, how far does it go?" (requires $d = rt$)

HumanEval & MBPP (code generation)

• What: 164 (HumanEval) / 974 (MBPP) Python programming problems with unit tests
• Why it matters: Executable = objective correctness check
• Limitation: Tests algorithmic coding, not software engineering (design, debugging, collaboration)

MT-Bench (Multi-Turn Benchmark)

• What: 80 multi-turn conversations across 8 categories (writing, reasoning, math, coding, roleplay, extraction, STEM, humanities)
• Why it matters: Tests conversational ability + context tracking (real-world chat use)
• Evaluation: GPT-4 as judge, rates1-10 per turn

AlpacaEval

• What: 805 diverse instructions, compares model vs reference (GPT-4-Turbo)
• Why it matters: Measures instruction-following + user preference
• Metric: Win rate (% of time model preferred over baseline)

Part 4: Pitfalls & Limitations

Comprehensive Example: Evaluating a New Model

Scenario: You've fine-tuned LaMA-3-70B on medical data. How do you evaluate it?

Step 1: Benchmark suite

• MedQA (medical licensing exam questions): 78% (baseline: 72%, GPT-4: 86%)
  • Interpretation: Above baseline, but not SOTA for medical reasoning
• PubMedQA (yes/no/maybe on research abstracts): 81% (baseline: 76%, GPT-4: 84%)
  • Interpretation: Solid biomedical understanding
• HumanEval (general coding): 45% (baseline: 42%)
  • Interpretation: Coding not harmed, but no improvement

Step 2: LM-as-judge (GPT-4) on500 custom medical queries

• Prompt: "You are an expert physician. Rate the response for accuracy, clinical relevance, and safety (1-10 each)."
• Results: Accuracy 7.8, Relevance 8.3, Safety 9.1(baseline: 7.2, 7.9, 8.8)
  • Interpretation: Fine-tuning improved relevance + safety, small accuracy gain

Step 3: Human expert review (50 samples)

• Agreement with GPT-4 judge: Cohen's $\kappa = 0.68$ (substantial)
• Catch cases where judge was wrong: 4 responses had factual errors judge missed
  • Interpretation: LLM judge reliable for screening, but critical decisions need humans

Step 4: A/B test in simulated clinic

• 20 physicians use model vs baseline for 100 patient questions
• Preference: 65% prefer fine-tuned model (AlpacaEval-style win rate)
• Qualitative: "More specific drug recommendations, fewer generic responses"

Conclusion: Model is production-ready for physician assistant role (not autonomous diagnosis). Benchmarks showed capability, judge showed quality improvement, humans validated safety.

Active Recall Practice

Connections

• Chain-of-Thought Prompting: GSM8K/MATH benchmarks test CoT reasoning ability
• RLHF (Reinforcement Learning from Human Feedback): Human preference data trains reward models, similar to LM-judge setup
• Prompt Engineering Best Practices: Benchmark prompts must be carefully designed (zero-shot vs few-shot affects scores)
• RAG (Retrieval-Augmented Generation): Evaluation needs benchmarks for retrieval quality (recall, precision) + generation quality (faithfulness)
• Alignment Tax: High benchmark scores don't guarantee safety—need alignment-specific evals (TruthfulQA, BBQ bias benchmark)
• Few-Shot Learning: MT-Bench tests in-context learning across conversation turns
• Constitutional AI: Uses model-as-judge internally to filter harmful outputs during training

#flashcards/ai-ml

What are the three core challenges that make LM evaluation harder than traditional ML evaluation? :: 1) Capability diversity (models do many tasks), 2) Output subjectivity (no single "correct" answer for open-ended tasks), 3) Human evaluation cost (can't scale expert review to millions of outputs)

Derive the Pass@k formula from first principles :: Pass@k = P(≥1 correct in k samples) = 1 - P(all k wrong) = 1 - (1-p)^k where p is the probability a single sample is correct, estimated as c/n (correct solutions / total problems)

Concept Map

Hinglish (regional understanding)