Hallucination mitigation: Worked examples — AI-ML deep dive

This page is a drill hall. The parent note Hallucination mitigation gave you the formulas; here we throw every kind of input at them so you never meet a case you haven't already solved by hand.

Before we compute anything, let us name the two number-machines we will use, in plain words, so no symbol appears unearned.

Definition The two machines on this page

Machine 1 — HallucinationRate. You take an answer, chop it into small statements that are each true-or-false on their own (call each an atomic claim). Count how many there are ( $N$ ) and how many are supported by trusted evidence ( $C$ ). Then $HallucinationRate = \frac{N - C}{N}, Faithfulness = \frac{C}{N} .$ $N - C$ is simply "the ones that failed". Dividing by $N$ turns a raw count into a fraction between $0$ and $1$ , so a 6-claim answer and a 60-claim answer live on the same ruler.

Machine 2 — Selective Risk & Coverage. The model attaches a confidence $s$ (a number, higher = "I'm surer") to each answer, and only speaks when $s \geq τ$ , where $τ$ (Greek "tau") is a threshold you choose — a cutoff line. Then $Coverage (τ) = \frac{# { answered }}{# { total }}, Risk (τ) = \frac{# { answered AND wrong }}{# { answered }} .$ Coverage = "what fraction did it dare to answer". Risk = "of the ones it answered, what fraction were wrong". Read $# {\dots}$ as "the number of things where …".

The symbol $\geq$ means "greater than or equal to". $Pr (\cdot)$ later means "probability of". That is the entire vocabulary; everything below is built from it.

The scenario matrix

Every worked example below is tagged with which cell it fills. The matrix is the promise: no cell is left un-shown.

Cell	What makes it tricky	Example
A. Ordinary case	some claims wrong, some right	Ex 1
B. Zero numerator ( $C = N$ )	nothing wrong → rate must be $0$	Ex 2
C. Full numerator ( $C = 0$ )	everything wrong → rate must be $1$	Ex 2
D. Degenerate input ( $N = 0$ )	no claims at all → division by zero!	Ex 3
E. Threshold sweep	move $τ$ across every data point	Ex 4
F. Limiting $τ$	$τ$ so low nothing is filtered / so high nothing answers	Ex 5
G. Ties at the boundary	a point sits exactly at $s = τ$	Ex 5
H. Voting / self-consistency	majority vote, incl. a tie	Ex 6
I. Real-world word problem	medical cost-of-error picks $τ$	Ex 7
J. Exam twist	faithfulness vs factuality trap	Ex 8

Machine 1 examples (HallucinationRate)

Worked example

Ex 1 — Cell A: the ordinary mixed answer Answer: "Mount Everest is the tallest mountain (✓), it lies on the Nepal–China border (✓), it is 9{,}000 m tall (✗, it's ~8{,}849 m), and it was first summited in 1953 (✓)."

Forecast: guess the rate before reading on — 4 claims, 1 wrong. Higher or lower than $\frac{1}{4}$ ?

List atomic claims. {tallest ✓, Nepal–China ✓, 9000 m ✗, 1953 ✓}. Why this step? One wrong fact should not condemn the whole sentence, so we split into independently checkable units (this is the FActScore idea).
Count. $N = 4$ , $C = 3$ . Why this step? The machine only needs these two counts.
Plug in. $HallucinationRate = \frac{N - C}{N} = \frac{4 - 3}{4} = 0.25.$

Verify: Faithfulness $= C / N = 3/4 = 0.75$ , and $0.25 + 0.75 = 1$ ✓ (the two must sum to one — every claim is either supported or not, no third option). Units: pure fraction, dimensionless ✓.

Worked example

Ex 2 — Cells B & C: the two extremes Part B (all correct): "Water is H₂O (✓); it freezes at 0 °C at sea level (✓); it boils at 100 °C at sea level (✓)." Part C (all wrong): "The Sun orbits the Earth (✗); the Moon is a star (✗); Pluto is the largest planet (✗)."

Forecast: what are the smallest and largest values this machine can ever return?

Part B counts: $N = 3, C = 3.$ Rate $= \frac{3 - 3}{3} = \frac{0}{3} = 0.$ Why this step? When nothing fails, the numerator $N - C$ is exactly zero, so the whole fraction collapses to $0$ — the machine's floor.
Part C counts: $N = 3, C = 0.$ Rate $= \frac{3 - 0}{3} = \frac{3}{3} = 1.$ Why this step? When everything fails, $C = 0$ so $N - C = N$ , and $N / N = 1$ — the machine's ceiling.

Verify: the output always lives in $[0, 1]$ because $0 \leq C \leq N$ forces $0 \leq N - C \leq N$ . Faithfulness for B is $1$ , for C is $0$ — mirror images ✓.

Worked example

Ex 3 — Cell D: the degenerate input ( $N = 0$ ) Answer: "Hmm, I don't have enough information to answer that." — a pure abstention. No factual claims are made.

Forecast: what does $\frac{N - C}{N}$ do when $N = 0$ ? Careful.

Try to plug in. $N = 0, C = 0 \Rightarrow \frac{0 - 0}{0} = \frac{0}{0}$ — undefined. Why this step? Division by zero is not "zero" or "one"; it is no answer at all. The formula was built assuming there is at least one claim to grade.
Fix by convention. When $N = 0$ we say HallucinationRate is not applicable (or define it as $0$ by fiat — "an answer that claims nothing cannot lie"). Why this step? An abstention should never be scored as a hallucination; that is the whole point of letting the model say "I don't know".

Verify: this matches the parent note's mitigation #6 — abstention is a safe output, so it must not count against factuality. A grader that blows up on $N = 0$ is a buggy grader ✓.

Machine 2 examples (Risk & Coverage)

We reuse one dataset for the threshold examples. Study the figure — it is the whole story on one number line.

Worked example

Ex 4 — Cell E: sweep the threshold across every point Forecast: as $τ$ climbs from $0$ to $1$ , does Risk go up or down? Does Coverage?

We evaluate $τ = 0.5$ and $τ = 0.7$ (the two thresholds that change the outcome). "Answer only if $s \geq τ$ ."

$τ = 0.5$ : kept = ${0.9, 0.8, 0.6, 0.55}$ (drop the $0.4$ ). Among kept, wrong = ${0.6}$ . Why this step? We literally apply the rule "speak iff $s \geq τ$ " and then look only at the survivors. $Coverage = \frac{4}{5} = 0.8, Risk = \frac{1}{4} = 0.25.$
$τ = 0.7$ : kept = ${0.9, 0.8}$ , both correct. Wrong among kept = $0$ . $Coverage = \frac{2}{5} = 0.4, Risk = \frac{0}{2} = 0.$
Read the trend. Raising $τ$ from $0.5 \to 0.7$ : Coverage $0.8 \to 0.4$ (down), Risk $0.25 \to 0$ (down). Why this step? This is the central trade-off — you buy lower risk by answering fewer questions.

Verify: the dropped points ( $0.4$ then $0.6, 0.55$ ) were exactly the low-confidence ones, and dropping a wrong point can only lower Risk. Coverage is monotone non-increasing in $τ$ — confirmed, $0.8 \geq 0.4$ ✓.

Worked example

Ex 5 — Cells F & G: limiting $τ$ and an exact tie Forecast: what are Coverage and Risk at the two ends $τ = 0$ and $τ = 1$ ? And what if $τ$ lands exactly on a point?

$τ = 0$ (floor). Every $s \geq 0$ , so all five are answered. Kept $=$ everyone; wrong $= {0.6, 0.4}$ . $Coverage = \frac{5}{5} = 1, Risk = \frac{2}{5} = 0.4.$ Why this step? $τ = 0$ = "always answer" — maximum coverage, and Risk equals the model's raw error rate.
$τ = 1$ (ceiling, and here no $s$ equals 1). No $s \geq 1$ (highest is $0.9$ ), so nothing is answered. $Coverage = \frac{0}{5} = 0, Risk = \frac{0}{0} (undefined — no answered questions) .$ Why this step? When the model never speaks, "error rate among the ones it answered" has no denominator. Convention: report Risk as undefined / not-applicable, not $0$ .
Exact tie, $τ = 0.6$ (Cell G). The rule uses $\geq$ , so the point $s = 0.6$ is kept. Kept $= {0.9, 0.8, 0.6}$ , wrong $= {0.6}$ . $Coverage = \frac{3}{5} = 0.6, Risk = \frac{1}{3} \approx 0.333.$ Why this step? Boundary behaviour is a decision, not an accident: " $\geq$ " includes the tie. Had we used strict " $>$ ", the $0.6$ would drop and Risk would fall to $0/2 = 0$ . Always state which inequality you use.

Verify: at $τ = 0$ Coverage is maximal ( $= 1$ ) ✓; at $τ = 1$ Coverage is minimal ( $= 0$ ) ✓; the tie at $0.6$ under " $\geq$ " keeps that wrong answer, raising Risk vs the strict version — consistent ✓.

Worked example

Ex 6 — Cell H: self-consistency voting, including a tie Part 1: Q "Who wrote Hamlet?" → 5 samples: Shakespeare, Shakespeare, Bacon, Shakespeare, Shakespeare. Part 2 (tie twist): Q with 4 samples: A, A, B, B.

Forecast: which answer survives voting in each part?

Part 1 tally. Shakespeare $= 4$ , Bacon $= 1$ . Majority $=$ Shakespeare ( $4/5 = 0.8$ of the vote). Why this step? True facts are attractors — the model keeps returning to them; the lone "Bacon" is an unstable fabrication, out-voted.
Part 2 tally. A $= 2$ , B $= 2$ — a dead tie ( $0.5$ each). No majority. Why this step? An even split means self-consistency gives no signal; the right move is to abstain or gather more samples, not to pick arbitrarily. (Ties are the degenerate case of voting.)

Verify: Part 1 vote share $4/5 = 0.8 > 0.5$ , so majority exists ✓. Part 2 shares are $2/4 = 0.5$ each — neither exceeds $0.5$ , so "strict majority" fails ✓, correctly flagging "abstain".

The hard cases: word problem & exam twist

Worked example

Ex 7 — Cell I: real-world word problem (medical cost of error) A triage assistant answers 100 patient questions. A wrong answer costs 50 units of harm; an abstention ("see a doctor") costs 1 unit of delay. Using the model's risk–coverage curve, two thresholds are available:

$τ_{low}$ : Coverage $= 0.90$ (90 answered), Risk $= 0.20$ .
$τ_{high}$ : Coverage $= 0.50$ (50 answered), Risk $= 0.02$ .

Forecast: which threshold gives less total harm? Guess before computing — high stakes favour caution.

Count wrong answers. Wrong $=$ Coverage $\times$ total $\times$ Risk.
- Low: $0.90 \times 100 \times 0.20 = 18$ wrong. Abstentions $= 100 - 90 = 10$ .
- High: $0.50 \times 100 \times 0.02 = 1$ wrong. Abstentions $= 100 - 50 = 50$ . Why this step? "Wrong count" = (how many answered) × (fraction of those that are wrong) — that is exactly Coverage × total × Risk.
Total harm = wrong·50 + abstentions·1.
- Low: $18 \times 50 + 10 \times 1 = 900 + 10 = 910.$
- High: $1 \times 50 + 50 \times 1 = 50 + 50 = 100.$ Why this step? We convert every outcome into the same unit (harm) so we can compare a risky-but-helpful policy against a cautious one.
Choose. $100 < 910$ , so pick $τ_{high}$ . Why this step? When a wrong answer is 50× costlier than a shrug, abstention wins — the risk–coverage curve is only half the story; the cost per error decides the operating point.

Verify: cheapen a wrong answer to cost 1 (equal to abstention) and recompute: Low $= 18 \times 1 + 10 = 28$ , High $= 1 \times 1 + 50 = 51$ ; now Low wins. So the chosen threshold flips with the cost ratio — the model is behaving sensibly ✓.

Worked example

Ex 8 — Cell J: exam twist (faithfulness vs factuality) Given context passage: "In our 2019 study, the drug reduced symptoms in 40% of patients." Model's answer: "The 2019 study found the drug reduced symptoms in 60% of patients, consistent with the well-known fact that it also cures headaches."

Forecast: classify each error. Is the "60%" a factuality or a faithfulness failure? What about "cures headaches"?

Split into atomic claims and pick the yardstick per claim. Why this step? Factuality is measured against the world; faithfulness against the given context. A single answer can violate both, so we grade each claim by the right ruler.
- "2019 study" — matches context ✓ (faithful).
- "60%" — context said 40% → faithfulness failure (contradicts the source). $N$ -wise: unsupported.
- "also cures headaches" — not in context, and false in the world → factuality failure (a fabricated external claim).
Score faithfulness (against context only). Claims checkable against context: {2019 ✓, 60% ✗}. So context-Faithfulness $= \frac{1}{2} = 0.5$ . Why this step? Faithfulness ignores world-facts and asks only "does the source support this?"
Score factuality (against world). Supported-by-world: {2019 ✓, headaches ✗} ⇒ and 60% is also world-wrong ⇒ of the 3 claims, 1 is true ⇒ HallucinationRate $= \frac{3 - 1}{3} \approx 0.667.$ Why this step? Different ruler, different number — this is exactly why the parent note warns "RAG eliminates hallucinations" is false: the model contradicted the context it was handed.

Verify: the "60%" claim fails both rulers (context and world), while "headaches" fails only the world ruler — showing faithfulness $\neq =$ factuality. Faithfulness $0.5$ and factual HallucinationRate $\approx 0.667$ are computed against different denominators (2 vs 3 claims), so they need not match ✓.

Recall

Recall Every cell in one breath

What does HallucinationRate return when every claim is right? ::: $0$ (numerator $N - C = 0$ ). What does it return when every claim is wrong? ::: $1$ ( $C = 0 \Rightarrow N / N$ ). What happens at $N = 0$ ? ::: Undefined ( $0/0$ ); treat an abstention as not-a-hallucination. As $τ \to 0$ , what are Coverage and Risk? ::: Coverage $\to 1$ ; Risk $\to$ raw error rate. As $τ \to$ above every $s$ , what is Risk? ::: Undefined — nothing answered, no denominator. A point sits exactly at $s = τ$ with rule " $\geq$ " — kept or dropped? ::: Kept. A 2–2 vote in self-consistency means? ::: No majority → abstain / sample more. What decides the operating threshold in the medical problem? ::: The cost ratio wrong:abstention, not the risk–coverage curve alone.

Prerequisite threads: RAG pipeline, Model calibration, Chain-of-thought reasoning, Evaluation metrics for LLMs.