6.4.15 · D5AI Safety & Alignment
Question bank — Responsible AI deployment practices
Before we start, one plain-words refresher so no term is used unearned:
Definition Vocabulary you'll need (click to expand)
- Blast radius ::: how many users/requests a failure can touch before it's caught. A 1% rollout shrinks the blast radius to 1% of traffic.
- Shadow mode ::: new model runs on real traffic but its output is thrown away; only logged for comparison. Users still see the old model.
- Concept drift ::: the relationship between input and correct answer changes (e.g. what counts as "spam" evolves).
- Data drift ::: the inputs change distribution even if the correct rule hasn't (e.g. more non-English names arrive).
- OOD ::: out-of-distribution — an input unlike anything the model saw in training (e.g. a blossoming tree to a classifier trained only on bare winter trees). The model has no good basis to answer, so it should be uncertain.
- Circuit breaker ::: an automatic switch that stops trusting the model and falls back to something safe when errors spike.
- HITL ::: Human-in-the-loop — a person reviews or approves certain predictions.
The whole pipeline on one picture
Before the traps, look at how the four safeguards fit together in a real-time request path. Follow the arrows: a request enters, the model answers, monitoring watches, and the circuit breaker / HITL decide when a human takes over.
The same four safeguards, staged over the timeline of a launch (internal → beta → 1% → 10% → 100%), form the defense-in-depth ladder — each layer catches what the one before missed:
One symbol at a time: the sample-size formula
Several traps below lean on the parent's rollout formula, so let's earn every letter first. Picture a stage as a bag of independent requests. Each request has a small chance of being a failure (a bad output we want to catch). We are willing to tolerate a probability (a small number like ) of missing the problem entirely — i.e. seeing zero failures even though the fault is real.
The picture of this formula — how required samples explode as the failure gets rarer — is worth seeing:
Why KL divergence, made concrete
The "why use KL divergence" trap is easier once you see two distributions with the same mean but different shape — the mean-only test says "no drift", KL says "big drift":
Sliding window vs all-time counter
And the circuit-breaker trap becomes obvious with a picture of a recovering model: an all-time counter never forgets and eventually trips forever; a sliding window tracks recent health and can reset:
A toy ROC curve for the 0.85 threshold
Finally, the "0.85 is standard" trap references ROC analysis. An ROC curve plots true-positive rate against false-positive rate as you slide the confidence threshold; the "right" operating point depends on your cost trade-off, not a magic number:
True or false — justify
A "staged rollout" means running each stage for a fixed calendar time regardless of traffic.
False. The statistic that matters is number of samples seen ( in the formula above), not days elapsed. A low-traffic day at 1% may not surface a rare failure, so gate criteria should be sample-based (or time plus minimum-sample), not clock-based alone.
Shadow mode has zero risk because users never see the shadow model.
Mostly true for user-facing risk, but not zero cost. The shadow model still consumes compute and can leak through logs/side effects (e.g. writing to a shared cache). "No user-visible output" is the guarantee, not "no risk at all."
A new model with a lower overall false-positive rate than the old one is always safe to promote.
False. A better average can hide a worse subgroup: the parent's example had 1.5% overall FPR but 8% on non-English names. Aggregate wins can mask fairness regressions.
If input drift (KL divergence) is near zero, model accuracy is guaranteed to hold.
False. Zero data drift does not rule out concept drift — the inputs look the same but the correct label rule changed. Drift on inputs is a leading indicator, not a completeness guarantee.
Rising average prediction entropy always means the model got worse.
False. Entropy rising means the model is less confident, which is often appropriate on genuinely ambiguous or novel (OOD) inputs. It's a signal to investigate, not proof of a bug — a wrongly over-confident model can even have low entropy.
A tripped circuit breaker means the model is broken.
False. It means recent predictions crossed a reliability threshold — which can be caused by an attack, drift, or an upstream data pipeline fault, not necessarily model rot. The breaker buys time to diagnose, it doesn't diagnose.
Human-in-the-loop review eliminates bias from the system.
False. Humans have their own biases and can rubber-stamp confident-looking AI outputs (automation bias). HITL redistributes and caps risk; it does not zero it.
Demographic parity within 5% across groups guarantees the model is fair.
False. Parity is one fairness definition among several (equalized odds, calibration, ...) and they can conflict. Passing one metric can mean failing another, so "fair" needs the metric named.
Spot the error
"We tested on 10,000 examples with no failures, so failures rarer than 1-in-10,000 won't happen in production."
The test set can only observe what it contains; absence of rare failures in 10k samples doesn't mean they're absent in 10M requests. A 0.01% rate is invisible in 10k but hits ~1,000 times/day at 10M/day — that's exactly why staged rollout exists.
"Our circuit breaker uses a fixed all-time error count, so it's stable."
A cumulative all-time count only ever grows and will trip eventually even after the model recovers (see figure s04). The breaker must use a sliding window (e.g. last 100 predictions) so it reflects recent health and can also reset.
"We set the confidence-routing threshold to 0.85 because that's the standard."
There is no universal threshold; 0.85 is an example. The right value is an operating point on the ROC curve (figure s05), chosen via validation analysis that trades automation rate against error cost for your application.
"We monitor overall accuracy, so we'll catch any degradation."
Overall accuracy needs ground-truth labels, which arrive late (or never) in production. Leading indicators — drift, entropy, human override rate — flag trouble before the accuracy drop is measurable.
"Adversarial inputs are a training problem, not a deployment problem."
Attackers act at deployment time on live traffic, exploiting the exact gap between test and production distributions (see 6.4.1-Adversarial-examples). Runtime monitoring and circuit breakers are deployment-side defenses precisely because retraining alone can't anticipate every attack.
"Once validation passes at 100% rollout, monitoring can be turned off."
Passing validation is a snapshot in time; the world keeps drifting. Monitoring is the ongoing pillar — turning it off removes your ability to detect drift, attacks, and silent failures.
"Higher latency is only a UX issue, not a safety issue."
Latency spikes can indicate infrastructure failure, retries, or an attack, and in time-critical settings (medical, fraud) slow answers can be unsafe answers. It's monitored on the same dashboard as safety metrics for that reason.
Why questions
Why does dividing by flip the inequality in the sample-size formula?
Because is negative for , and dividing an inequality by a negative number reverses its direction. That's why becomes .
Why use KL divergence rather than just comparing means of input features?
Means can be identical while whole shapes of the distribution differ (figure s03 shows two same-mean, different-shape distributions). KL compares the full distributions, measuring the extra bits needed to encode training data with a code tuned for production data.
Why is "human override rate" called the closest thing to ground truth on the dashboard?
Every other metric (drift, entropy, latency) is a proxy; the override rate is a real human judging a real prediction as wrong-enough-to-reject. A rising override rate is direct evidence the model's usefulness is slipping.
Why does defense-in-depth assume components will fail?
Because any single safeguard has a failure probability; layering independent safeguards makes the probability that all fail simultaneously tiny. It's the same logic as redundant systems on an aircraft — you design for the failure you can't prevent.
Why run shadow mode instead of jumping straight to a 1% rollout?
Shadow mode gives you production-scale, production-distribution data with zero user-visible impact, letting you catch subgroup and OOD failures before any real user is affected. A 1% rollout, by contrast, already exposes real users to the new model.
Why can a model pass every offline test and still fail in production?
Offline tests use a fixed, curated distribution; production traffic is adversarial, drifting, and full of long-tail inputs the test set never sampled. The mismatch between test distribution and live distribution is the core reason all three pillars exist.
Why tie the drift threshold to historical data rather than pick a round number?
Normal week-to-week traffic already fluctuates by some baseline amount; you must know that natural variation to set a threshold that fires on real drift and not on ordinary noise. A round number risks either constant false alarms or missed shifts.
Edge cases
What should the circuit breaker do the moment it trips — block all requests?
It should route to a safe fallback (simpler model, cached answer, human handoff, or a graceful "unavailable" message), not hard-crash. The point is degraded-but-safe service, keeping blast radius small while humans investigate.
A stage sees zero traffic during its window (holiday, outage). Does it pass?
No — zero samples means zero evidence, so a sample-based gate is not satisfied. Passing on an empty window would defeat the whole purpose of measuring failures before wider exposure.
Failure rate is essentially 0 (a truly safe model). What does the sample-size formula say?
As , , so : you'd need infinitely many samples to be confident you'd seen a failure. Correctly — you can never fully prove the absence of an arbitrarily rare failure by sampling alone.
Confidence threshold set to 0 for HITL routing — what happens?
Nothing is ever routed to humans (all predictions "exceed" 0), so the system is fully automated with no human safety net. Set it to 1 and everything routes to humans, collapsing to a fully manual system with no automation benefit.
All classes get equal predicted probability . What is the entropy, and what does it mean?
Entropy is at its maximum ( bits, or nats — pick a base and stay consistent): the model is maximally uncertain, effectively guessing. Sustained max-entropy predictions signal the model is out of its depth (OOD) on that input.
Two fairness metrics both must hold but mathematically cannot simultaneously. What do you do?
You cannot "pass" both; you must make an explicit, documented value choice about which harm matters more for this context, ideally under a governance process rather than silently. The impossibility itself must be surfaced, not hidden.
Drift is detected but you have no labels yet to confirm accuracy dropped. Deploy the retrain or wait?
Treat drift as a leading alarm: stage the retrained model behind shadow mode / a small rollout so you validate before committing, rather than blind-swapping. Waiting for labels means accepting possible silent harm; blind-swapping means shipping an unvalidated fix.
The new model is better on aggregate but worse for one tiny subgroup of 50 users. Ship it?
"Small group" is not a licence for harm — you evaluate the severity and reversibility of the harm to those users via subgroup analysis, not just the headcount. Often the fix is targeted (e.g. add underrepresented data) before promoting.
Recall One-line self-test
The three deployment pillars, in order? ::: Pre-deployment validation → runtime monitoring → human oversight architecture. The single design principle underneath all three? ::: Defense in depth — assume any one safeguard fails, so no single failure is catastrophic.