Responsible AI deployment practices: Worked examples

The scenario matrix

Before any example, let us list every cell a deployment-math problem can land in. Each row is a "shape" of input the formulas must survive.

#	Cell class	The tricky part	Covered by
A	Sample size for a rare failure ( $p$ tiny)	$ln (1 - p) \approx - p$ , huge $n$	Ex 1
B	Sample size for a common failure ( $p$ large)	small $n$ , no approximation allowed	Ex 2
C	Limiting case $p \to 0$	$n \to \infty$ — you can never be sure	Ex 3
D	KL divergence, normal shifted distributions	plug into the sum	Ex 4
E	KL divergence, identical distributions (degenerate)	must give exactly $0$	Ex 4
F	KL divergence, a zero bin in production	$lo g (finite /0) = + \infty$ trap	Ex 5
G	Prediction entropy: confident vs uncertain	which is "good"? sign of $lo g$	Ex 6
H	Entropy extremes: one-hot vs uniform	$0$ vs $lo g C$ bounds	Ex 6
I	Word problem: circuit breaker blast radius	translate story → counts	Ex 7
J	Word problem: staged rollout schedule in days	samples → traffic → time	Ex 8
K	Exam twist: confidence-routing automation rate	combine threshold + distribution	Ex 9

Every symbol below is re-earned. If you have not met $ln$ , KL divergence $D_{K L}$ , or entropy $H$ before, read the "First, what the symbol means" box the first time it appears.

Warm-up: the two symbols we lean on

Figure — Responsible AI deployment practices

Look at the curve above: it dives to $- \infty$ near $x = 0$ and crosses zero at $x = 1$ . Both the sample-size math and the KL/entropy math live entirely on the left half ( $0 < x < 1$ ), where $ln$ is negative — keep that picture in mind, it explains every sign flip below.

Group A–C · Sample-size math (how long to run a stage)

Recall the parent formula: to be $1 - α$ confident of catching at least one failure of rate $p$ , $n \geq \frac{l n ( α )}{l n ( 1 - p )} .$ Both $ln (α)$ and $ln (1 - p)$ are negative (both arguments are between 0 and 1), so their ratio is positive — good, $n$ is a count.

Worked example Ex 1 — Cell A: a

rare failure (p tiny) A self-driving perception model fails to see a pedestrian once in every 5,000 frames ( $p = 0.0002$ ). How many production frames must a 1% stage observe to be 99% confident of catching at least one such miss ( $α = 0.01$ )?

Forecast: guess — hundreds? thousands? tens of thousands?

Write the numbers: $p = 0.0002$ , $α = 0.01$ . Why this step? Name every symbol before touching the formula — the contract.
Numerator: $ln (0.01) = - 4.6052$ . Why? This is "how deep into confidence we want" — more confidence (smaller $α$ ) → more negative → bigger $n$ .
Denominator: $ln (1 - 0.0002) = ln (0.9998) = - 0.00020002$ . Why? For tiny $p$ , $ln (1 - p) \approx - p$ ; this is why rare events force huge $n$ — you divide by a near-zero number.
Divide: $n \geq \frac{- 4.6052}{- 0.00020002} \approx 23, 024$ frames. Why? Negative over negative = positive count.

Verify: plug back — probability of missing it in 23,024 frames is $(0.9998)^{23024} \approx 0.0100 = α$ . ✓ Units: a pure count of frames. Sanity: rarer event → larger $n$ , matches the $\approx 1/ p$ intuition ( $1/0.0002 = 5000$ , times $\approx 4.6$ for 99%).

Worked example Ex 2 — Cell B: a

common failure (p large) An early prototype hallucinates in 20% of medical answers ( $p = 0.2$ ). How many outputs must reviewers read to be 95% sure ( $α = 0.05$ ) of seeing at least one hallucination?

Forecast: big number like Ex 1, or tiny?

Write the numbers: $p = 0.2$ , $α = 0.05$ . Why this step? Same contract as Ex 1 — name every symbol in plain numbers before it touches the formula, so no quantity sneaks in undefined.
Numerator: $ln (0.05) = - 2.9957$ . Why this step? $α = 0.05$ is our tolerated 5% chance of missing the issue; its $ln$ is negative because $0.05$ is between 0 and 1 (see figure s01), and it sets how demanding our confidence is.
Denominator: $ln (1 - 0.2) = ln (0.8) = - 0.22314$ . Why no approximation here? $p = 0.2$ is not small, so $ln (1 - p) \neq = - p$ ; using $- 0.2$ would give the wrong answer. Cell B exists precisely to stop you reflexively approximating.
$n \geq \frac{- 2.9957}{- 0.22314} \approx 13.43 \Rightarrow 14$ outputs (round up — you cannot read a fraction of an output). Why round up? $\geq$ means we need at least this many; 13 would fall short of 95%.

Verify: $(0.8)^{14} = 0.0440 \leq 0.05$ ✓ but $(0.8)^{13} = 0.0550 > 0.05$ ✗ — so 14 is genuinely the smallest integer that works. Sanity: common failures need only a handful of samples. ✓

Worked example Ex 3 — Cell C: the limiting case

$p \to 0$ Question: what happens to $n$ as the failure gets arbitrarily rare, $p \to 0$ ?

Denominator behaviour: $ln (1 - p) \to ln (1) = 0^{-}$ (approaches zero from below, stays negative). Why? Look again at figure s01 — as the argument $\to 1$ , $ln \to 0$ .
Numerator $ln (α)$ is a fixed negative number.
Fixed negative $\div$ (negative approaching 0) $\to + \infty$ .

Conclusion: $p \to 0 lim n = + \infty$ . You can never fully guarantee catching an infinitely rare failure. This is the mathematical face of the parent's "curse of the long tail" — see 6.4.1-Adversarial-examples, where attackers deliberately manufacture near-zero-probability inputs.

Verify (numeric feel): at $α = 0.05$ , $p = 0.001 \Rightarrow n \approx 2995.7$ ; $p = 0.0001 \Rightarrow n \approx 29956.1$ . Ten times rarer → about ten times more samples. The $\approx 1/ p$ growth is unbounded. ✓

Group D–F · KL divergence (is production drifting from training?)

Figure s02 draws exactly the two distributions of Ex 4: the black bars are $P_{train}$ , the red bars are the shifted $P_{prod}$ . Notice bucket 2 has equal-height bars — that is the term that contributes 0 to the sum. KL divergence adds up, bin by bin, how far the red bar has strayed from the black one; the two big gaps at buckets 1 and 3 are where the drift number comes from.

Worked example Ex 4 — Cells D & E: normal shift, and the identical (degenerate) case

A feature falls into 3 buckets. Training saw $P_{train} = (0.5, 0.3, 0.2)$ .

Part D — production shifted to $P_{prod} = (0.2, 0.3, 0.5)$ (buckets 1 and 3 swapped popularity). Compute $D_{K L} (P_{train} ∥ P_{prod})$ in bits.

Term 1: $0.5 lo g_{2} \frac{0.5}{0.2} = 0.5 \times 1.3219 = 0.6610$ .
Term 2: $0.3 lo g_{2} \frac{0.3}{0.3} = 0.3 \times 0 = 0$ . Why zero? Bucket 2 didn't move — no surprise, exactly matching the equal-height red and black bars in figure s02.
Term 3: $0.2 lo g_{2} \frac{0.2}{0.5} = 0.2 \times (- 1.3219) = - 0.2644$ . Why is a term negative? Individual terms can be negative when $P < Q$ ; only the total is guaranteed $\geq 0$ .
Sum: $0.6610 + 0 - 0.2644 = 0.3966$ bits.

Part E — no drift at all: if $P_{prod} = P_{train}$ , every ratio is $\frac{P}{P} = 1$ , every $lo g_{2} = 0$ , so $D_{K L} = 0$ exactly. This is the degenerate case and the reason KL is a valid drift alarm: zero means "identical, do nothing."

Verify: Part D $\approx 0.3966$ bits $> 0$ ✓ (drift detected — below a 0.5 threshold, so watch but don't retrain yet). Part E $= 0$ ✓. Both consistent with $D_{K L} \geq 0$ always.

Worked example Ex 5 — Cell F: the zero-bin trap

Production suddenly serves inputs your training set never contained: $P_{train} = (0.5, 0.5, 0)$ but $P_{prod} = (0.4, 0.4, 0.2)$ . Naively compute $D_{K L} (P_{prod} ∥ P_{train})$ .

Forecast: a normal small number, or something explosive?

Term 3: $0.2 lo g_{2} \frac{0.2}{0}$ . Why does this break? Dividing by $0$ gives $+ \infty$ , and $lo g_{2} (+ \infty) = + \infty$ . So $D_{K L} = + \infty$ .
Interpretation: the model is being asked to explain data it has literally zero preparation for — infinite surprise. This is a hard stop, not a warning.
Engineering fix (why it exists): add tiny smoothing $ϵ$ to every bin so no denominator is exactly zero. With $ϵ = 0.001$ the $\infty$ becomes a large-but-finite spike that still fires the alert.

Verify: with the reference bin $= 0$ , the term $\to \infty$ ; the direction $D_{K L} (P_{prod} ∥ P_{train})$ matters — swap the arguments and the offending bin sits in the first factor as $0 \times lo g_{2} (0/ finite) = 0$ , which is finite. The order of KL's arguments changes whether you catch this failure. Always put the reference (training) distribution as $Q$ (denominator) so novel inputs blow up loudly. ✓

Group G–H · Prediction entropy (is the model confident?)

Figure s03 shows the three predictions of Ex 6 as bars of entropy in bits. The two dashed guide-lines are the bounds we will prove: the floor at $0$ (perfect certainty) and the ceiling at $lo g_{2} 4 = 2$ (total confusion, red bar). Every real prediction lands somewhere between them — reading the height of a bar tells you at a glance how nervous the model is.

Worked example Ex 6 — Cells G & H: confident vs uncertain, and the two extremes

A 4-class image classifier. Compare three outputs (using $lo g_{2}$ , so $H$ is in bits, bounded by $lo g_{2} 4 = 2$ ).

(G, confident): $p = (0.9, 0.05, 0.03, 0.02)$ .

Write one term per class: $H = - (0.9 lo g_{2} 0.9 + 0.05 lo g_{2} 0.05 + 0.03 lo g_{2} 0.03 + 0.02 lo g_{2} 0.02)$ . Why one term per class? Entropy sums a contribution from every possible outcome; a class the model almost ignores ( $0.02$ ) still adds a little uncertainty, so we must include all four.
Evaluate each $lo g_{2}$ : $lo g_{2} 0.9 = - 0.152$ , $lo g_{2} 0.05 = - 4.322$ , $lo g_{2} 0.03 = - 5.059$ , $lo g_{2} 0.02 = - 5.644$ . Why are they all negative? Each probability is between 0 and 1 (figure s01), so its $lo g_{2}$ is negative; the leading minus sign in $H$ will flip the total back to positive.
Multiply and add: $H = 0.1368 + 0.2161 + 0.1518 + 0.1129 = 0.6176$ bits. Low → healthy. Why "healthy"? A small $H$ means the model concentrated its belief on one class — it is confident, exactly what we want in production.

(H-min, one-hot extreme): $p = (1, 0, 0, 0)$ . Using the convention $0 lo g_{2} 0 = 0$ , every term vanishes: $H = 0$ . Perfect certainty. This is the lower bound.

(H-max, uniform extreme): $p = (0.25, 0.25, 0.25, 0.25)$ .

$H = - 4 \times 0.25 lo g_{2} 0.25 = - 1 \times (- 2) = 2$ bits. Why exactly 2? $lo g_{2} 0.25 = - 2$ and $lo g_{2} C = lo g_{2} 4 = 2$ — for $C$ equally-likely classes $H = lo g_{2} C$ , the upper bound. Total confusion.

Verify: confident $\approx 0.6176$ bits, sitting comfortably between the bounds $0 \leq H \leq 2$ . ✓ If your live average climbs from $0.6$ toward $2$ , the model is heading for uniform-guessing — retrain, exactly the winter→spring blossom story in the parent.

Group I–J · Word problems (story → numbers)

Worked example Ex 7 — Cell I: circuit-breaker blast radius

A spam filter uses a circuit breaker with error threshold $0.10$ and window $= 100$ . An attacker floods it with 1,000 adversarial emails; the model's confidence drops below $0.7$ on exactly 15% of them (these count as "errors"). How many emails get misclassified before the breaker trips, and what is the blast radius saved?

Forecast: does it trip early or let most through?

Translate the story into counts: flood size $= 1000$ emails, error fraction $= 0.15$ , window $= 100$ , trip threshold $= 0.10$ . Why this step? Same naming contract — pin down every quantity in plain numbers before reasoning, so no count is invented later.
Errors accumulate in a sliding window of 100. Once the window is full of representative traffic, its error rate reads $0.15$ . Why 0.15? 15% of a steady stream is 15 errors per 100 — that is the window's error rate.
Compare to the trip threshold: $0.15 > 0.10$ , so the breaker trips as soon as the window fills — after the first $\approx 100$ emails. Why 100, not sooner? The rate is computed over len(recent_errors); the window must be populated for the estimate to be trustworthy.
Count damage before the trip: emails misclassified $\approx 100 \times 0.15 = 15$ . Remaining $1000 - 100 = 900$ get routed to human reviewers (safe fallback). Why multiply? The error fraction applied to the emails seen before tripping gives the actual misclassifications leaked.

Verify: blast radius = 15 bad emails instead of $1000 \times 0.15 = 150$ across the whole flood — a $10 \times$ reduction, matching the parent's "100 instead of 1,000 misclassified" framing (there they counted flagged emails; here we count misclassifications). Units: emails. ✓ See 6.3.2-MLOps-principles for where such breakers live in a pipeline.

Worked example Ex 8 — Cell J: staged-rollout schedule in days

You need $n = 2996$ samples to catch a $p = 0.001$ failure at 95% confidence (parent's own example). Your product gets 8,000 users/day, and the 1% stage sends 1% of that traffic to the new model. How many days must the stage run?

Forecast: hours, days, or weeks?

Daily samples reaching the new model: $8000 \times 0.01 = 80$ users/day. Why multiply? 1% traffic routing means only 1% of users are the "trials" in the sample-size math.
Days needed: $\frac{2996}{80} = 37.45$ days $\Rightarrow$ 38 days (round up). Why round up? Fewer days = fewer samples = below the required confidence.

Verify: $38 \times 80 = 3040 \geq 2996$ ✓ while $37 \times 80 = 2960 < 2996$ ✗ — so 38 is genuinely the smallest whole number of days that clears the target. Design lesson: if 38 days is too slow, raise the traffic fraction — bump to 10% and $n$ is reached in $⌈ 2996/800 ⌉ = 4$ days. This is the levers the parent's "1% → 10% → 100%" ladder gives you. Compare with 5.2.4-A-B-testing for the same sample-size logic in experiment design.

Group K · Exam-style twist

Worked example Ex 9 — Cell K: automation rate from confidence routing

Exam twist: The parent routes predictions below confidence $0.85$ to humans (see also 6.4.12-Explainability-methods for why humans need explanations to review them). Suppose the model's top-confidence values, over many requests, are uniformly spread between 0.6 and 1.0. (a) What fraction is handled automatically? (b) If human review costs $2 per case and you get 50,000 requests/day, what is the daily review bill?

Forecast: guess the automation percentage first.

(a) Automatic means confidence $\geq 0.85$ . On the interval $[0.6, 1.0]$ (width $0.4$ ), the automatic slice is $[0.85, 1.0]$ (width $0.15$ ). Why lengths? "Uniform" means probability = length fraction. Automation rate $= \frac{1.0 - 0.85}{1.0 - 0.6} = \frac{0.15}{0.40} = 0.375$ , i.e. 37.5% automatic.
Human fraction $= 1 - 0.375 = 0.625$ , i.e. 62.5% routed to people. Why subtract from 1? Every request is either automated or human-reviewed; the two fractions must sum to the whole.
(b) Human cases/day $= 50000 \times 0.625 = 31250$ . Cost $=31250\times \$ 2 = $62{,}500$/day. Why multiply twice? First scale total traffic by the human fraction to get case count, then by the per-case price to get money.

Verify: $0.375 + 0.625 = 1$ ✓ (fractions sum to one). Cost check: $50000 \times 0.625 \times 2 = 62500$ ✓, units = dollars/day. Insight: raising the threshold to $0.9$ shrinks the automatic slice to $\frac{0.1}{0.4} = 25%$ , increasing the human bill — the safety/cost trade-off the parent hinted at with "tune via ROC." Fairness note: check the human-routed cases aren't skewed by group (6.4.8-Fairness-metrics, 6.4.14-AI-governanceframeworks).

Recall Self-test (reveal after answering)

Rare failures force which limit for $n$ ? ::: $n \to \infty$ as $p \to 0$ (Ex 3). Why must training be the denominator $Q$ in $D_{K L}$ ? ::: So a novel (zero-in-training) production bin makes the ratio blow up to $+ \infty$ and fires a loud alert (Ex 5). A 4-class model outputs $(0.25, 0.25, 0.25, 0.25)$ ; its entropy in bits is? ::: $lo g_{2} 4 = 2$ bits — maximum confusion (Ex 6). With 8,000 users/day at 1% routing, a 2,996-sample stage takes how long? ::: 38 days (Ex 8). Uniform confidence on $[0.6, 1.0]$ , threshold 0.85 → automation rate? ::: 37.5% (Ex 9).