Responsible AI deployment practices: Visual walkthrough

We will earn every symbol. By the end you will understand this line completely:

$n \geq \frac{l n ( α )}{l n ( 1 - p )}$

...and you will know why every letter is where it is.

Step 1 — What "a failure rate" even means

WHAT. We start with a single number: $p$ . It is the failure probability of one request.

WHY. Before we can ask "how many requests to catch a failure", we must have a crisp handle on how often a failure happens. That handle is $p$ .

PICTURE. Imagine a long strip of tiny boxes, one box per request. Most are green (success). A rare few are red (failure). The red fraction of the whole strip is $p$ .

Look at the strip: only a sliver is red. That sliver is $p$ . The whole point of a staged rollout is that if we only look at a few boxes, we might see zero red — and wrongly conclude the model is safe.

Step 2 — The chance a single request does NOT fail

WHAT. If failing has probability $p$ , then not failing has probability $1 - p$ .

WHY. Our goal is to catch at least one failure. That is awkward to count directly (it could be one, two, three... failures). It is far easier to count the opposite: seeing zero failures. So we first need the probability that one request succeeds — that's the building block for "everything succeeds".

PICTURE. Take the strip from Step 1 and collapse it to a single spinner: a wheel where a red wedge of size $p$ means "fail" and the green rest of size $1 - p$ means "succeed".

The green wedge is fat (because $p$ is tiny), so one spin almost always lands green. That "almost always" is exactly what makes rare failures dangerous — and exactly why one request tells you almost nothing.

Step 3 — Why we multiply: the chance ALL n requests succeed

WHAT. Spin the wheel $n$ times (send $n$ requests). The probability that every single one lands green is:

$(1 - p) \times (1 - p) \times \dots \times (1 - p) = (1 - p)^{n}$

WHY multiply, not add? Because the requests are independent — the outcome of request 1 doesn't change request 2. When you want "this AND this AND this" for independent events, probabilities multiply. (Adding would answer "this OR this", a different question.)

PICTURE. A branching tree: at each request the path forks green (success) or red (failure). The single all-green path down the left edge has probability $(1 - p)^{n}$ — and it shrinks fast as the tree gets deeper.

Notice: even though one spin is almost surely green, demanding a long chain of greens gets less and less likely. That shrinking is our friend — it means with enough requests, "all green" becomes rare, so seeing all green becomes real evidence of safety.

Step 4 — Turning our wish into an inequality

WHAT. We do not want to miss the failure. Let $α$ be the risk we tolerate of missing it — the probability we're willing to accept that all $n$ requests come up green even though the model is buggy.

We insist that the "all green" probability be no bigger than $α$ :

$chance we see zero failures (1 - p)^{n} \leq chance we’re willing to be fooled α$

WHY. If the chance of "all green despite a bug" is squeezed below our tolerance $α$ , then seeing all green is good enough evidence to trust the model. This inequality is the statement of our goal, in symbols.

PICTURE. A vertical bar. The tall part is $α$ (our tolerance ceiling). The curve $(1 - p)^{n}$ slides down as $n$ grows. We need $n$ big enough that the curve dips under the ceiling.

The yellow ceiling is fixed. The blue curve falls with more requests. We want the smallest $n$ where blue drops below yellow.

Step 5 — Why the logarithm? Unsticking $n$ from the exponent

WHAT. Our unknown $n$ is trapped up in the exponent of $(1 - p)^{n}$ . We need a tool that pulls an exponent down to ground level. That tool is the logarithm.

WHY this tool and not another? We could not solve $(1 - p)^{n} \leq α$ for $n$ with ordinary algebra — dividing or square-rooting won't free an exponent. The logarithm is the one operation built to answer "what power?", so it is the natural key to a locked exponent. Taking $ln$ of both sides is legal because $ln$ is increasing (bigger input → bigger output), so it preserves the $\leq$ direction... for now.

Apply $ln$ to both sides:

$ln ((1 - p)^{n}) \leq ln (α) ⟹ now freed n \cdot ln (1 - p) \leq ln (α)$

PICTURE. The curve $y = ln (x)$ : it drags large exponents down onto a straight-line playground where $n$ is just a plain multiplier.

Step 6 — The sign trap: why the inequality FLIPS

WHAT. We now divide both sides by $ln (1 - p)$ to isolate $n$ . But watch the sign of $ln (1 - p)$ .

Since $p$ is between $0$ and $1$ , the number $1 - p$ is less than 1. And $ln$ of any number less than 1 is negative:

$ln (1 - p) < 0 (because 0 < 1 - p < 1)$

Dividing $n \cdot ln (1 - p) \leq ln (α)$ by the negative quantity $ln (1 - p)$ flips $\leq$ into $\geq$ :

$n \geq \frac{ln ( α )}{ln ( 1 - p )}$

Every symbol earned:

$ln (α)$ — a negative number (since $α < 1$ ), "how strict am I?"
$ln (1 - p)$ — a negative number (since $1 - p < 1$ ), "how rare is one failure?"
negative ÷ negative = positive, so $n$ comes out sensibly positive.
$\geq$ — we need at least this many; more is fine, fewer is not.

WHY. The $\geq$ is the whole punchline: it tells us the minimum sample size. Forget the flip and you'd compute $n \leq$ something — concluding you need fewer requests the stricter you get, which is nonsense.

PICTURE. A number line showing both $ln$ values living in negative territory, and the division landing $n$ safely on the positive side, with an arrow marking "you need this many or more".

Step 7 — Plugging in real numbers

WHAT. Detect a $0.1%$ failure rate with $95%$ confidence: $p = 0.001$ , $α = 0.05$ .

$n \geq \frac{l n ( 0.05 )}{l n ( 1 - 0.001 )} = \frac{l n ( 0.05 )}{l n ( 0.999 )} = \frac{- 2.9957}{- 0.0010005} \approx 2995.6$

Round up (you can't run a fraction of a request, and $\geq$ forbids rounding down):

$n = 2996 requests.$

WHY. This is the number the parent note quotes. It means: run the new model on ~3000 real requests. If none fail, you can say with $95%$ confidence its failure rate is below $0.1%$ . At 1000 users/day that's 3 days of 1% traffic.

PICTURE. The falling curve $(1 - p)^{n}$ with a vertical marker at $n \approx 2996$ where it crosses the $α = 0.05$ ceiling.

Step 8 — The edge cases (never let the reader hit an unshown scenario)

WHAT & WHY & PICTURE, case by case:

Worked example Every corner of the formula

Case $p \to 0$ (the flaw is vanishingly rare). $ln (1 - p) \to 0^{-}$ (a tiny negative). Dividing by something near zero blows $n$ up to infinity. Meaning: an ultra-rare failure needs a mountain of traffic to catch — sometimes practically undetectable by sampling alone. This is exactly the long tail the parent warns about.

Case $p \to 1$ (the model fails almost always). $ln (1 - p) \to - \infty$ , so $n \to 0$ . Meaning: a model that fails constantly reveals itself in a handful of requests. Obvious — and the math agrees.

Case $α \to 1$ (I tolerate any miss). $ln (α) \to ln (1) = 0$ , so $n \to 0$ . Meaning: if you don't care about catching the bug, you need no requests. Useless in practice, but internally consistent.

Case $α \to 0$ (I refuse to ever be fooled). $ln (α) \to - \infty$ , so $n \to \infty$ . Meaning: absolute certainty requires infinite testing. There is no free lunch — this is why we accept a small $α$ instead of demanding zero.

Degenerate $p = 0$ exactly. $ln (1 - 0) = ln (1) = 0$ → division by zero, undefined. Interpretation: a truly perfect model can never be "caught failing" because it never fails; the question itself dissolves.

The figure sweeps $p$ across its range and shows $n$ exploding as $p \to 0$ and collapsing as $p \to 1$ — the whole behavioural map on one plot.

The one-picture summary

One flow, left to right: one spinner ( $1 - p$ green) → multiply for $n$ spins ( $(1 - p)^{n}$ ) → cap it below tolerance ( $\leq α$ ) → take $ln$ to free $n$ → divide (sign flips!) → the final $n \geq \frac{ln α}{ln ( 1 - p )}$ .

Recall Feynman retelling — say it like you'd tell a friend

I've got a new model. It messes up rarely — call that rare-mistake chance $p$ . So it gets things right with chance $1 - p$ each try. If I test it $n$ times and it's genuinely buggy, the chance it happens to look perfect (all green) is $1 - p$ multiplied by itself $n$ times, which is $(1 - p)^{n}$ . I don't want to be fooled more than, say, $5%$ of the time — that $5%$ is $α$ . So I demand $(1 - p)^{n} \leq α$ . The trouble: $n$ is stuck up in the exponent. The logarithm is the one tool that yanks an exponent down to a plain multiplier, so I take $ln$ of both sides and get $n ln (1 - p) \leq ln α$ . But $1 - p$ is under $1$ , so its log is negative — and dividing by a negative flips the sign — giving $n \geq ln α / ln (1 - p)$ . Both logs are negative, so their ratio is a nice positive number: the minimum requests I need. For a $1$ -in- $1000$ bug caught with $95%$ confidence, that's about $2996$ requests. And the edges make sense: rarer bug → need way more tests; stricter confidence → need way more tests; there's never a free lunch.

Recall

What role does the exponent $n$ play in $(1 - p)^{n}$ ? ::: It counts how many independent requests must all succeed in a row — how many green spins we demand. Why do we take a logarithm at all? ::: It is the only tool that pulls the unknown $n$ down from the exponent, turning $(1 - p)^{n}$ into $n ln (1 - p)$ so we can solve for $n$ . Why does the inequality flip to $\geq$ ? ::: Because $ln (1 - p)$ is negative (as $1 - p < 1$ ), and dividing an inequality by a negative number reverses its direction. As $p \to 0$ , what happens to the required $n$ , and what does it mean? ::: $n \to \infty$ — an extremely rare failure needs enormous traffic to detect; the long-tail problem. For $p = 0.001, α = 0.05$ , what is $n$ ? ::: About $2996$ requests (round up from $2995.6$ ).

See also: 6.3.2-MLOps-principles (where this gates a staged rollout pipeline), 5.2.4-A-B-testing (the same sampling logic for comparing two models), 6.4.1-Adversarial-examples (the rare failures this math tries to catch), and the parent Responsible AI deployment practices.