Responsible AI deployment practices: Worked examples

6.4.15 · D3 · AI-ML › AI Safety & Alignment › Responsible AI deployment practices

The scenario matrix

Kisi bhi example se pehle, chalte hain har cell list karte hain jahan ek deployment-math problem land kar sakti hai. Har row ek "shape" hai input ki jo formulas ko survive karni chahiye.

#	Cell class	Tricky part	Covered by
A	Sample size ek rare failure ke liye ( $p$ tiny)	$ln (1 - p) \approx - p$ , bahut bada $n$	Ex 1
B	Sample size ek common failure ke liye ( $p$ large)	chhota $n$ , approximation allowed nahi	Ex 2
C	Limiting case $p \to 0$	$n \to \infty$ — tum kabhi sure nahi ho sakte	Ex 3
D	KL divergence, normal shifted distributions	sum mein plug karo	Ex 4
E	KL divergence, identical distributions (degenerate)	exactly $0$ dena chahiye	Ex 4
F	KL divergence, production mein ek zero bin	$lo g (finite /0) = + \infty$ ka trap	Ex 5
G	Prediction entropy: confident vs uncertain	kaun sa "good" hai? $lo g$ ka sign	Ex 6
H	Entropy extremes: one-hot vs uniform	$0$ vs $lo g C$ bounds	Ex 6
I	Word problem: circuit breaker blast radius	story → counts mein translate karo	Ex 7
J	Word problem: staged rollout schedule in days	samples → traffic → time	Ex 8
K	Exam twist: confidence-routing automation rate	threshold + distribution combine karo	Ex 9

Neeche har symbol dobara earn kiya jayega. Agar tumne $ln$ , KL divergence $D_{K L}$ , ya entropy $H$ pehle nahi dekha, toh pehli baar "First, what the symbol means" box padho.

Warm-up: do symbols jinpe hum ziada rely karte hain

Figure — Responsible AI deployment practices

Upar wala curve dekho: $x = 0$ ke paas $- \infty$ mein jaata hai aur $x = 1$ par zero cross karta hai. Sample-size math aur KL/entropy math dono poori tarah left half ( $0 < x < 1$ ) par jeete hain, jahan $ln$ negative hota hai — yeh picture dimag mein rakho, yeh neeche ke har sign flip ko explain karta hai.

Group A–C · Sample-size math (ek stage kitni der tak run karein)

Parent formula yaad karo: $1 - α$ confidence ke saath at least ek failure of rate $p$ pakadne ke liye, $n \geq \frac{l n ( α )}{l n ( 1 - p )} .$ $ln (α)$ aur $ln (1 - p)$ dono negative hain (dono arguments 0 aur 1 ke beech hain), toh unka ratio positive hai — acha hai, $n$ ek count hai.

Worked example Ex 1 — Cell A: ek

rare failure (p tiny) Ek self-driving perception model har 5,000 frames mein ek baar pedestrian dekhna miss karta hai ( $p = 0.0002$ ). 1% stage ke production mein kitne frames observe karne honge taaki 99% confident rahein ki at least ek aisa miss pakad saken ( $α = 0.01$ )?

Forecast: guess karo — hundreds? thousands? tens of thousands?

Numbers likho: $p = 0.0002$ , $α = 0.01$ . Yeh step kyun? Formula chhoone se pehle har symbol ko naam do — yeh contract hai.
Numerator: $ln (0.01) = - 4.6052$ . Kyun? Yeh hai "hum kitna deep confidence chahte hain" — zyada confidence (chhota $α$ ) → aur negative → bada $n$ .
Denominator: $ln (1 - 0.0002) = ln (0.9998) = - 0.00020002$ . Kyun? Tiny $p$ ke liye, $ln (1 - p) \approx - p$ ; isliye rare events huge $n$ force karte hain — tum ek near-zero number se divide kar rahe ho.
Divide karo: $n \geq \frac{- 4.6052}{- 0.00020002} \approx 23, 024$ frames. Kyun? Negative divided by negative = positive count.

Verify: plug back karo — 23,024 frames mein isko miss karne ki probability $(0.9998)^{23024} \approx 0.0100 = α$ hai. ✓ Units: frames ka pure count. Sanity: rare event → bada $n$ , $\approx 1/ p$ intuition se match karta hai ( $1/0.0002 = 5000$ , times $\approx 4.6$ for 99%).

Worked example Ex 2 — Cell B: ek

common failure (p large) Ek early prototype 20% medical answers mein hallucinate karta hai ( $p = 0.2$ ). Reviewers ko kitne outputs padhne honge taaki 95% sure ho sakein ( $α = 0.05$ ) ki at least ek hallucination dikhega?

Forecast: Ex 1 jaisa bada number, ya tiny?

Numbers likho: $p = 0.2$ , $α = 0.05$ . Yeh step kyun? Same contract Ex 1 jaisa — formula mein daalne se pehle har symbol ko plain numbers mein pin down karo, taaki koi quantity undefined na rahe.
Numerator: $ln (0.05) = - 2.9957$ . Yeh step kyun? $α = 0.05$ hamaari tolerated 5% chance hai issue miss karne ki; iska $ln$ negative hai kyunki $0.05$ 0 aur 1 ke beech hai (figure s01 dekho), aur yeh set karta hai ki humari confidence kitni demanding hai.
Denominator: $ln (1 - 0.2) = ln (0.8) = - 0.22314$ . Yahaan approximation kyun nahi? $p = 0.2$ small nahi hai, toh $ln (1 - p) \neq = - p$ ; $- 0.2$ use karna galat answer dega. Cell B exactly isliye exist karta hai taaki tum reflexively approximate na karo.
$n \geq \frac{- 2.9957}{- 0.22314} \approx 13.43 \Rightarrow 14$ outputs (round up karo — output ka fraction nahi padh sakte). Round up kyun? $\geq$ ka matlab hai hume at least itne chahiye; 13 se 95% nahi milega.

Verify: $(0.8)^{14} = 0.0440 \leq 0.05$ ✓ lekin $(0.8)^{13} = 0.0550 > 0.05$ ✗ — toh 14 genuinely sabse chhota integer hai jo kaam karta hai. Sanity: common failures ke liye sirf chand samples chahiye. ✓

Worked example Ex 3 — Cell C: limiting case

$p \to 0$ Sawal: jab failure arbitrarily rare hoti jaati hai, $p \to 0$ , toh $n$ ka kya hota hai?

Denominator ka behaviour: $ln (1 - p) \to ln (1) = 0^{-}$ (zero ke neeche se approach karta hai, negative rehta hai). Kyun? Figure s01 phir se dekho — jab argument $\to 1$ , $ln \to 0$ .
Numerator $ln (α)$ ek fixed negative number hai.
Fixed negative $\div$ (negative approaching 0) $\to + \infty$ .

Conclusion: $p \to 0 lim n = + \infty$ . Tum kabhi guarantee nahi kar sakte ki infinitely rare failure pakad loge. Yeh parent ke "curse of the long tail" ka mathematical face hai — 6.4.1-Adversarial-examples dekho, jahan attackers deliberately near-zero-probability inputs banate hain.

Verify (numeric feel): $α = 0.05$ par, $p = 0.001 \Rightarrow n \approx 2995.7$ ; $p = 0.0001 \Rightarrow n \approx 29956.1$ . Failure das guna rare → lagbhag das guna zyada samples. $\approx 1/ p$ growth unbounded hai. ✓

Group D–F · KL divergence (kya production training se drift kar rahi hai?)

Figure s02 exactly Ex 4 ki do distributions draw karta hai: black bars $P_{train}$ hain, red bars shifted $P_{prod}$ hain. Dekho bucket 2 mein equal-height bars hain — yeh woh term hai jo sum mein 0 contribute karta hai. KL divergence add karta hai, bin by bin, ki red bar black bar se kitna door gaya hai; buckets 1 aur 3 par do bade gaps hain jahan se drift number aata hai.

Worked example Ex 4 — Cells D & E: normal shift, aur identical (degenerate) case

Ek feature 3 buckets mein aati hai. Training ne dekha $P_{train} = (0.5, 0.3, 0.2)$ .

Part D — production shifted hokar $P_{prod} = (0.2, 0.3, 0.5)$ ho gayi (buckets 1 aur 3 ki popularity swap ho gayi). $D_{K L} (P_{train} ∥ P_{prod})$ bits mein compute karo.

Term 1: $0.5 lo g_{2} \frac{0.5}{0.2} = 0.5 \times 1.3219 = 0.6610$ .
Term 2: $0.3 lo g_{2} \frac{0.3}{0.3} = 0.3 \times 0 = 0$ . Zero kyun? Bucket 2 move nahi hua — koi surprise nahi, exactly figure s02 mein equal-height red aur black bars se match karta hai.
Term 3: $0.2 lo g_{2} \frac{0.2}{0.5} = 0.2 \times (- 1.3219) = - 0.2644$ . Ek term negative kyun hai? Individual terms negative ho sakte hain jab $P < Q$ ho; sirf total $\geq 0$ guaranteed hota hai.
Sum: $0.6610 + 0 - 0.2644 = 0.3966$ bits.

Part E — koi drift nahi: agar $P_{prod} = P_{train}$ , har ratio $\frac{P}{P} = 1$ hai, har $lo g_{2} = 0$ , toh $D_{K L} = 0$ exactly. Yeh degenerate case hai aur reason hai ki KL ek valid drift alarm hai: zero matlab "identical, kuch mat karo."

Verify: Part D $\approx 0.3966$ bits $> 0$ ✓ (drift detect hua — 0.5 threshold se neeche, toh watch karo lekin abhi retrain mat karo). Part E $= 0$ ✓. Dono $D_{K L} \geq 0$ hamesha se consistent hain.

Worked example Ex 5 — Cell F: zero-bin trap

Production suddenly aise inputs serve karne lagti hai jo tumhare training set mein kabhi nahi the: $P_{train} = (0.5, 0.5, 0)$ lekin $P_{prod} = (0.4, 0.4, 0.2)$ . Naively $D_{K L} (P_{prod} ∥ P_{train})$ compute karo.

Forecast: ek normal small number, ya kuch explosive?

Term 3: $0.2 lo g_{2} \frac{0.2}{0}$ . Yeh kyun toot jaata hai? $0$ se divide karne par $+ \infty$ milta hai, aur $lo g_{2} (+ \infty) = + \infty$ . Toh $D_{K L} = + \infty$ .
Interpretation: model se aise data explain karne ki expect ki ja rahi hai jiske liye uski literally zero preparation hai — infinite surprise. Yeh ek hard stop hai, sirf warning nahi.
Engineering fix (kyun exist karta hai): har bin mein tiny smoothing $ϵ$ add karo taaki koi denominator exactly zero na ho. $ϵ = 0.001$ ke saath $\infty$ ek large-but-finite spike ban jaati hai jo alert fire karna continue karta hai.

Verify: reference bin $= 0$ ke saath, term $\to \infty$ ; $D_{K L} (P_{prod} ∥ P_{train})$ direction matter karta hai — arguments swap karo aur offending bin pehle factor mein $0 \times lo g_{2} (0/ finite) = 0$ ho jaata hai, jo finite hai. KL ke arguments ka order change karta hai ki tum yeh failure pakadoge ya nahi. Hamesha reference (training) distribution ko $Q$ (denominator) rakhO taaki novel inputs loudly blow up karein. ✓

Group G–H · Prediction entropy (kya model confident hai?)

Figure s03 Ex 6 ki teen predictions ko bits mein entropy bars ki tarah dikhata hai. Do dashed guide-lines woh bounds hain jo hum prove karenge: $0$ par floor (perfect certainty) aur $lo g_{2} 4 = 2$ par ceiling (total confusion, red bar). Har real prediction inka between land hoti hai — ek bar ki height dekh ke tum ek nazar mein samajh sakte ho ki model kitna nervous hai.

Worked example Ex 6 — Cells G & H: confident vs uncertain, aur do extremes

Ek 4-class image classifier. Teen outputs compare karo ( $lo g_{2}$ use karo, toh $H$ bits mein hai, $lo g_{2} 4 = 2$ se bounded).

(G, confident): $p = (0.9, 0.05, 0.03, 0.02)$ .

Har class ke liye ek term likho: $H = - (0.9 lo g_{2} 0.9 + 0.05 lo g_{2} 0.05 + 0.03 lo g_{2} 0.03 + 0.02 lo g_{2} 0.02)$ . Ek term per class kyun? Entropy har possible outcome ka contribution sum karta hai; ek class jise model almost ignore karta hai ( $0.02$ ) bhi thodi uncertainty add karti hai, toh saanon chaar include karna zaroori hai.
Har $lo g_{2}$ evaluate karo: $lo g_{2} 0.9 = - 0.152$ , $lo g_{2} 0.05 = - 4.322$ , $lo g_{2} 0.03 = - 5.059$ , $lo g_{2} 0.02 = - 5.644$ . Sab negative kyun hain? Har probability 0 aur 1 ke beech hai (figure s01), toh uska $lo g_{2}$ negative hai; $H$ mein leading minus sign total ko wapas positive flip karega.
Multiply karke add karo: $H = 0.1368 + 0.2161 + 0.1518 + 0.1129 = 0.6176$ bits. Low → healthy. "Healthy" kyun? Chhota $H$ matlab model ne apna belief ek class par concentrate kiya — yeh confident hai, exactly wahi jo hum production mein chahte hain.

(H-min, one-hot extreme): $p = (1, 0, 0, 0)$ . Convention $0 lo g_{2} 0 = 0$ use karte hue, har term vanish ho jaati hai: $H = 0$ . Perfect certainty. Yeh lower bound hai.

(H-max, uniform extreme): $p = (0.25, 0.25, 0.25, 0.25)$ .

$H = - 4 \times 0.25 lo g_{2} 0.25 = - 1 \times (- 2) = 2$ bits. Exactly 2 kyun? $lo g_{2} 0.25 = - 2$ aur $lo g_{2} C = lo g_{2} 4 = 2$ — $C$ equally-likely classes ke liye $H = lo g_{2} C$ , upper bound hai. Total confusion.

Verify: confident $\approx 0.6176$ bits, bounds $0 \leq H \leq 2$ ke beech comfortably baitha hai. ✓ Agar tumhara live average $0.6$ se $2$ ki taraf climb karne lage, model uniform-guessing ki taraf ja raha hai — retrain karo, exactly parent mein winter→spring blossom story ki tarah.

Group I–J · Word problems (story → numbers)

Worked example Ex 7 — Cell I: circuit-breaker blast radius

Ek spam filter circuit breaker use karta hai error threshold $0.10$ aur window $= 100$ ke saath. Ek attacker uspe 1,000 adversarial emails flood karta hai; model ka confidence exactly 15% emails par $0.7$ se neeche drop ho jaata hai (yeh "errors" count hote hain). Breaker trip hone se pehle kitne emails misclassify ho jaate hain, aur blast radius kya save hoti hai?

Forecast: kya yeh jaldi trip karta hai ya zyaadatar through let karta hai?

Story ko counts mein translate karo: flood size $= 1000$ emails, error fraction $= 0.15$ , window $= 100$ , trip threshold $= 0.10$ . Yeh step kyun? Same naming contract — reasoning karne se pehle har quantity ko plain numbers mein pin down karo, taaki baad mein koi count invented na ho.
Errors ek sliding window of 100 mein accumulate hote hain. Jab window representative traffic se bhar jaati hai, uski error rate $0.15$ read hoti hai. 0.15 kyun? Steady stream ka 15% matlab 100 mein 15 errors — yeh window ki error rate hai.
Trip threshold se compare karo: $0.15 > 0.10$ , toh breaker trip karta hai jaise hi window fill hoti hai — pehle $\approx 100$ emails ke baad. 100, pehle nahi kyun? Rate len(recent_errors) par compute hoti hai; estimate trustworthy hone ke liye window populated honi chahiye.
Trip se pehle damage count karo: emails misclassified $\approx 100 \times 0.15 = 15$ . Baaki $1000 - 100 = 900$ human reviewers ke paas route ho jaate hain (safe fallback). Multiply kyun? Trip hone se pehle dekhe gaye emails par error fraction apply karne se actual misclassifications niklaate hain jo leak hue.

Verify: blast radius = 15 bad emails instead of $1000 \times 0.15 = 150$ pure flood mein — ek $10 \times$ reduction, parent ke "100 instead of 1,000 misclassified" framing se match karta hai (wahan unhone flagged emails count kiye the; yahaan hum misclassifications count kar rahe hain). Units: emails. ✓ 6.3.2-MLOps-principles dekho ki aisa breaker pipeline mein kahan rehta hai.

Worked example Ex 8 — Cell J: staged-rollout schedule in days

Tumhe $n = 2996$ samples chahiye ek $p = 0.001$ failure ko 95% confidence par pakadne ke liye (parent ka apna example). Tumhare product ko 8,000 users/day milte hain, aur 1% stage naye model par 1% traffic bhejti hai. Stage kitne days run karni chahiye?

Forecast: hours, days, ya weeks?

Daily samples jo naye model tak pahunchte hain: $8000 \times 0.01 = 80$ users/day. Multiply kyun? 1% traffic routing matlab sirf 1% users sample-size math mein "trials" hain.
Days needed: $\frac{2996}{80} = 37.45$ days $\Rightarrow$ 38 days (round up karo). Round up kyun? Kam days = kam samples = required confidence se neeche.

Verify: $38 \times 80 = 3040 \geq 2996$ ✓ jabki $37 \times 80 = 2960 < 2996$ ✗ — toh 38 genuinely sabse chhota whole number of days hai jo target clear karta hai. Design lesson: agar 38 days bahut slow hai, traffic fraction badha do — 10% karo aur $n$ $⌈ 2996/800 ⌉ = 4$ days mein reach ho jaata hai. Yahi levers hain jo parent ka "1% → 10% → 100%" ladder deta hai. Same sample-size logic ke liye 5.2.4-A-B-testing compare karo experiment design mein.

Group K · Exam-style twist

Worked example Ex 9 — Cell K: confidence routing se automation rate

Exam twist: Parent confidence $0.85$ se neeche ki predictions humans ko route karta hai (yeh bhi dekho 6.4.12-Explainability-methods ki humans ko review karne ke liye explanations kyun chahiye). Maano model ki top-confidence values, bahut saare requests par, uniformly 0.6 aur 1.0 ke beech spread hain. (a) Kitna fraction automatically handle hota hai? (b) Agar human review ka cost $2 per case hai aur tumhare paas 50,000 requests/day hain, toh daily review bill kya hai?

Forecast: pehle automation percentage guess karo.

(a) Automatic matlab confidence $\geq 0.85$ . Interval $[0.6, 1.0]$ (width $0.4$ ) par, automatic slice $[0.85, 1.0]$ (width $0.15$ ) hai. Lengths kyun? "Uniform" matlab probability = length fraction. Automation rate $= \frac{1.0 - 0.85}{1.0 - 0.6} = \frac{0.15}{0.40} = 0.375$ , yaani 37.5% automatic.
Human fraction $= 1 - 0.375 = 0.625$ , yaani 62.5% logon ke paas route hota hai. 1 se subtract kyun? Har request ya toh automated hai ya human-reviewed; dono fractions milke pura sum hone chahiye.
(b) Human cases/day $= 50000 \times 0.625 = 31250$ . Cost $=31250\times \$ 2 = $62{,}500$/day. Do baar multiply kyun? Pehle total traffic ko human fraction se scale karo case count paane ke liye, phir per-case price se money paane ke liye.

Verify: $0.375 + 0.625 = 1$ ✓ (fractions sum to one). Cost check: $50000 \times 0.625 \times 2 = 62500$ ✓, units = dollars/day. Insight: threshold $0.9$ tak raise karne se automatic slice $\frac{0.1}{0.4} = 25%$ tak shrink ho jaati hai, human bill badh jaata hai — yahi safety/cost trade-off hai jo parent ne "tune via ROC" mein hint kiya tha. Fairness note: check karo ki human-routed cases kisi group ki taraf skewed toh nahi hain (6.4.8-Fairness-metrics, 6.4.14-AI-governanceframeworks).

Recall Self-test (answer karne ke baad reveal karo)

Rare failures $n$ ke liye kaunsi limit force karti hain? ::: $n \to \infty$ as $p \to 0$ (Ex 3). Training $D_{K L}$ mein denominator $Q$ kyun honi chahiye? ::: Taaki ek novel (zero-in-training) production bin ka ratio $+ \infty$ tak blow up ho jaaye aur loud alert fire kare (Ex 5). Ek 4-class model $(0.25, 0.25, 0.25, 0.25)$ output karta hai; bits mein uski entropy kya hai? ::: $lo g_{2} 4 = 2$ bits — maximum confusion (Ex 6). 8,000 users/day par 1% routing ke saath, ek 2,996-sample stage kitna time leti hai? ::: 38 days (Ex 8). Uniform confidence on $[0.6, 1.0]$ , threshold 0.85 → automation rate kya hai? ::: 37.5% (Ex 9).