Responsible AI deployment practices: Exercises — Hinglish deep dive

6.4.15 · D4 · AI-ML › AI Safety & Alignment › Responsible AI deployment practices

Is page ke ideas 6.4.1-Adversarial-examples, 6.4.8-Fairness-metrics, 6.4.12-Explainability-methods, 6.3.2-MLOps-principles, 5.2.4-A-B-testing, aur 6.4.14-AI-governanceframeworks se liye gaye hain. Hinglish version chahiye? Yeh page Hinglish mein →.

Symbols Jo Tumhe Chahiye Honge (ek baar build kiye, har jagah use hote hain)

Definition Vocabulary, Plain Words Mein

$p$ — failure rate: requests ka woh fraction jo galat jaate hain. Agar $1000$ mein se $1$ request fail hoti hai, toh $p = 0.001$ .
$n$ — samples ki sankhya (requests) jo hum ek rollout stage mein observe karte hain.
$α$ — hamara acceptable miss chance: woh probability jo hum accept karne ko taiyar hain ki ek bhi failure nahi dikhegi jabki model faulty hai. $α = 0.05$ matlab "main $5%$ chance accept karta hoon ki mujhe bewakoof banaya jaaye."
$ln$ — natural logarithm (base $e$ ): yeh jawab deta hai " $e$ ko kis power pe raise karein ki yeh number mile?" Yeh products ko sums mein badal deta hai, aur isliye yeh tab appear hota hai jab hum kai independent probabilities ko multiply karte hain.
Entropy aur KL ke liye Base convention: is poore page mein, $lo g$ jo $H$ aur $D_{K L}$ ke andar hai woh $lo g_{2}$ (base-2) hai, toh dono quantities bits mein measured hain. Yeh choice yahan ek baar state ki gayi hai aur consistently use hoti hai; agar hum $ln$ use karte, toh numbers constant factor $ln 2 \approx 0.693$ se rescale hote lekin comparisons identical rehte.
$D_{K L} (P ∥ Q) = \sum_{i} P_{i} lo g_{2} \frac{P _{i}}{Q _{i}}$ — KL divergence: do distributions kitni alag hain iska ek-number score. Zero matlab identical; zyada matlab "woh duniya jo main ab dekh raha hoon ( $Q$ ) ab woh duniya nahi lagti jis par maine train kiya tha ( $P$ )." Edge case: agar kisi bin mein $Q_{i} = 0$ ho jabki $P_{i} > 0$ ho, toh term $lo g_{2} (P_{i} /0)$ $+ \infty$ tak blow up karta hai, toh $D_{K L}$ undefined (infinite) hai. Practice mein hum Laplace smoothing apply karte hain — $Q$ ke har bin mein ek tiny $ϵ$ (jaise $1 0^{- 6}$ ) add karo aur renormalize karo — taki koi bin kabhi exactly zero na ho.
$H = - \sum_{i} p_{i} lo g_{2} p_{i}$ — entropy (bits mein): ek probability guess kitni spread out hai. Ek confident guess (ek class near $1$ ) ka $H$ low hota hai; ek "mujhe bilkul idea nahi" guess (sab classes equal) ka $H$ high hota hai. Edge case: $p_{i} = 0$ wala term $0$ contribute karta hai (limit $p lo g p \to 0$ as $p \to 0$ ), toh empty classes simply skip ho jaati hain.

Yeh definitions saamne rakh lo. Neeche har problem inhi tools mein se ek hai jo ek clear sawal ka jawab de rahi hai.

Level 1 — Recognition

Exercise 1.1

Har safeguard ko us failure se match karo jo woh prevent karta hai: (a) Shadow mode, (b) Circuit breaker, (c) Staged rollout, (d) Confidence-based routing. Failures: (i) ek rare edge case jo ek saath sare users ko wipe out kar de, (ii) users ka broken naye model ko bilkul dekhna, (iii) adversarial inputs ki ek burst jo hazaaron bad decisions mein cascade ho jaaye, (iv) ek low-confidence prediction jo bina kisi human check ke act ho jaaye.

Recall Solution

(a) Shadow mode → (ii). Naya model parallel mein run karta hai lekin uska output kabhi users ko nahi dikhaya jaata, toh failure ka zero user impact hota hai.
(b) Circuit breaker → (iii). Yeh low-confidence/error signals ki burst ke baad "trip" karta hai, blast radius ko cap karta hai (upar define kiya gaya).
(c) Staged rollout → (i). Audience ko slowly grow karne ka matlab hai ki ek rare edge case pehle ek small population ko hit karta hai.
(d) Confidence-based routing → (iv). Below-threshold predictions ko human ke paas bhej diya jaata hai. Yeh kyun matter karta hai: yeh teen pillars hain — pre-deployment (c, a), runtime (b), oversight (d) — har ek alag failure mode ko catch karta hai. Neeche diya diagram har safeguard ko deployment timeline par place karta hai taki tum dekh sako ki har ek kahan blast radius ko shrink karta hai.

Figure — Responsible AI deployment practices

Diagram deployment lifecycle ke saath left se right padhta hai. Staged rollout (lavender) aur shadow mode (mint) pehle act karte hain jab users expose hote hain; widening funnel dikhata hai ki audience sirf tab grow hoti hai jab har gate pass ho jaata hai. Circuit breaker (coral) runtime par baitha hai, failures ki spike ko clamp karta hai. Confidence routing (butter) uncertain cases ko ek human ke paas peel off karta hai. Shaded "blast radius" band har baar shrink hoti hai jab ek safeguard intervene karta hai.

Exercise 1.2

Formula $n \geq \frac{ln ( α )}{ln ( 1 - p )}$ mein, kaun sa symbol "real failure miss karne ka acceptable chance" hai? Kaun sa "failure rate" hai?

Recall Solution

$α$ = acceptable miss chance (jaise $0.05$ ).
$p$ = failure rate jise hum pakadna chahte hain (jaise $0.001$ ). $n$ (zaroorat ke samples) tab badhta hai jab $p$ shrink hota hai (rarer bugs ko zyada looks chahiye) aur jab $α$ shrink hota hai (zyada certainty ke liye zyada looks chahiye).

Level 2 — Application

Exercise 2.1

Tumhe suspect hai ki failure rate $p = 0.002$ ( $0.2%$ ) hai. Tum chahte ho ki ise catch karne ka $90%$ chance ho ( $α = 0.10$ ). Stage ko kitne samples $n$ observe karne chahiye?

Recall Solution

WHAT: $n \geq \frac{ln α}{ln ( 1 - p )}$ mein plug karo. YEH FORMULA KYUN: $n$ independent tries mein zero failures ka chance $(1 - p)^{n}$ hai; hum ise force karte hain ki woh at most $α$ ho, phir $n$ ke liye solve karte hain. $n \geq \frac{l n ( 0.10 )}{l n ( 1 - 0.002 )} = \frac{- 2.3026}{- 0.0020020} \approx 1150.2$ Round up karo (tum fractional request observe nahi kar sakte): $n = 1151$ samples. Agar is stage par daily volume $500$ users hai, toh woh $⌈ 1151/500 ⌉ = 3$ din hai.

Exercise 2.2

$3$ classes wala ek image classifier ek request par probabilities $p = (0.7, 0.2, 0.1)$ output karta hai. Prediction entropy $H = - \sum_{i} p_{i} lo g_{2} p_{i}$ compute karo (base-2, toh answer "bits" mein hai — Symbols box mein fix ki gayi convention match karta hai).

Recall Solution

ENTROPY YAHAN KYUN: hum ek single number chahte hain ki "yeh guess kitna unsure hai?" Ek confident guess ka score low hona chahiye. $H = - (0.7 lo g_{2} 0.7 + 0.2 lo g_{2} 0.2 + 0.1 lo g_{2} 0.1)$ $= - (0.7 (- 0.5146) + 0.2 (- 2.3219) + 0.1 (- 3.3219))$ $= 0.3602 + 0.4644 + 0.3322 = 1.1568 bits$ Neeche diya figure padho is number ko ek scale par place karne ke liye: mint bar ek totally sure guess $(1, 0, 0)$ hai jiska $H = 0$ hai; coral bar total confusion $(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ hai jiska $H = lo g_{2} 3 \approx 1.585$ bits hai; hamara butter bar $1.157$ par unke beech mein hai — moderately confident. Visual concretely dikhata hai ki production mein rising average entropy (Exercise 3.2) ek alarm kyun hai: bars rightward slide ho rahe hain confusion ki taraf.

Exercise 2.3

$3$ bins par training feature histogram: $P = (0.5, 0.3, 0.2)$ . Is hafte ka production histogram: $Q = (0.2, 0.3, 0.5)$ . $D_{K L} (P ∥ Q) = \sum_{i} P_{i} lo g_{2} \frac{P _{i}}{Q _{i}}$ compute karo (base-2, bits — same convention).

Recall Solution

KL YAHAN KYUN: yeh score karta hai "ek model jo $P$ par tuned hai woh $Q$ milne par kitna surprised hoga." Zyada = zyada drift. $D_{K L} = 0.5 lo g_{2} \frac{0.5}{0.2} + 0.3 lo g_{2} \frac{0.3}{0.3} + 0.2 lo g_{2} \frac{0.2}{0.5}$ $= 0.5 (1.3219) + 0.3 (0) + 0.2 (- 1.3219)$ $= 0.6610 + 0 - 0.2644 = 0.3966 bits$ Note karo ki middle bin zero contribute karta hai (woh move nahi kiya) aur shrinking bin ek negative term contribute karta hai — lekin growing bin ise outweigh karta hai, toh total positive hai, jaise KL hamesha hota hai. Edge-case reminder: yahan har $Q_{i} > 0$ hai, toh sum well-defined hai. Agar koi production bin empty hota ( $Q_{i} = 0$ ) jabki training ne use dekha tha ( $P_{i} > 0$ ), toh woh term $lo g_{2} (P_{i} /0) = + \infty$ hota aur $D_{K L}$ undefined hota — isliye real monitors $Q$ ko pehle Laplace-smooth karte hain (Symbols box dekho).

Level 3 — Analysis

Exercise 3.1

Ek hafte ke shadow mode mein ek content-moderation model:

Old model false-positive rate: $2.0%$ .
New model false-positive rate: $1.5%$ .
Lekin new-model false positives split: non-English names par $8%$ , English names par $2%$ .

Kya tumhe new model ko $1%$ traffic par promote karna chahiye? Fairness lens use karke justify karo.

Recall Solution

Nahi. Aggregate metric ( $1.5% < 2.0%$ ) ek disparity hide kar raha hai. Demographic-parity idea use karke: false-positive rate group ke hisaab se differ karta hai. Ratio $8%/2% = 4 \times$ — ek $4$ -to- $1$ gap jo "within $5%$ " parity gate se kaafi zyada hai. Diagnosis: English-biased training data ne model ko sikhaya "uncommon character sequences $\Rightarrow$ spam," toh Xiang ya Aditya jaisi names false positives trigger karti hain. Ek lower average error ek subgroup ki keemat par buy kiya gaya tha. Action: promote mat karo. Protected features ke roop mein multilingual name dictionaries ke saath training par return karo; kisi bhi staged rollout se pehle group ke hisaab se sliced shadow comparison re-run karo.

Exercise 3.2

Ek image classifier ki average entropy readings: Train $0.30$ , Week 1 $0.32$ , Week 4 $0.55$ (bits). Trend ko interpret karo aur likely cause ka naam batao. Kaun sa specific runtime metric ise confirm karega?

Recall Solution

Rising entropy = model average pe kam sure ho raha hai = woh aise inputs se mil raha hai jo uski training mein nahi the (out-of-distribution). Week 1 ka tiny rise ( $+ 0.02$ ) noise hai; Week 4 ki $0.55$ tak jump (almost doubled) ek red flag hai 🚨. Likely cause: ek distribution shift — jaise seasonal change (training mein bare winter trees vs. abhi blossoming spring trees). Yeh data drift hai. Confirming metric: input features par $D_{K L} (P_{train} ∥ P_{prod})$ compute karo. Agar woh bhi apne threshold se spike kare, toh drift confirm ho jaata hai aur retrain justified hai. Entropy batata hai model unsure hai; KL batata hai inputs actually change ho gaye — tum chaahte ho dono agree karein pehle action lene se.

Level 4 — Synthesis

Exercise 4.1

Ek spam classifier ke liye ek circuit breaker design karo. Requirements:

Ek prediction ko "error" treat karo agar uski confidence $< 0.7$ ho.
Trip (open) karo jab last $w in d o w = 100$ predictions mein error rate $t h r es h o l d = 0.10$ se exceed kare.
Attack ke under, $1000$ adversarial emails ki ek stream mein $15%$ emails ki confidence $< 0.7$ hai.

(a) Kitni emails ke baad breaker trip karta hai? (b) Trip se pehle kitni bad emails users tak pahunchti hain, aur baaki ko humans ke paas route karke kitni bachayi jaati hain? (c) Full state machine specify karo — startup, reset, aur cooldown including.

Recall Solution

(a) Trip kab hota hai? — explicit answer. Breaker last $100$ predictions mein error rate ko $t h r es h o l d = 0.10$ ke against compare karta hai, yani woh tab trip karta hai jab trailing $100$ mein se $10$ se zyada errors hon ( $11$ ya zyada). $15%$ steady error rate ke saath errors $0.15$ per email ki rate se aate hain. $11$ errors accumulate karne ke liye hum expect karte hain $11/0.15 \approx 73.3$ emails ki zaroorat hogi — lekin window ko contain karna hoga $100$ predictions jab tak ek "last-100" rate define bhi ho, aur jab tak $100$ vi email process hoti hai window mein $\approx 0.15 \times 100 = 15$ errors hain, jo already $> 10$ hai. Toh steady-rate assumption ke under, pehle woh moment jab ek full trailing window exist karta hai ( $n = 100$ ) woh already $\approx 15$ errors dikhata hai aur trip ho jaata hai. Explicit answer: breaker $100$ vi email par trip karta hai (woh earliest point jab ek full $100$ -prediction window exist karti hai, us point par woh already over threshold hai).

(b) Blast radius — explicit numbers.

Trip se pehle model ne actually jo emails process kiye: $100$ .
Un mein se misclassified (low-confidence errors served): $15% \times 100 =$ $15$ bad emails users tak pahunchi.
Trip ke baad human reviewers ko route ki gayi emails: $1000 - 100 =$ $900$ emails (sab remaining traffic fallback par jaata hai).
Un $900$ mein se prevented bad emails: $15% \times 900 \approx$ $135$ likely errors avoid kiye gaye. Contrast: bina breaker ke, sab $1000$ emails model ko hit karti hain aur $\approx 150$ misclassified hoti hain. Breaker served errors ko $\approx 150$ se ghataa kar $15$ tak le aata hai — blast radius mein $10 \times$ reduction.

(c) Full state machine — startup, cooldown, aur ek explicit RESET. Neeche diya diagram har state aur transition dikhata hai; har ek yahan spell out hai.

CLOSED (normal): model predictions serve karta hai; har ek last $100$ events ki sliding window mein log hota hai.
Startup rule (problem statement se, koi invented hyperparameter nahi): trip condition yeh hai ki "last $100$ predictions mein error rate $0.10$ se exceed kare." Literally liya jaaye, woh rate sirf tab defined hai jab $100$ predictions exist karti hain. Isliye breaker window full hone se pehle trip nahi kar sakta ( $n \geq 100$ ) — startup behaviour simply hai "CLOSED raho jab tak trailing- $100$ window exist na ho." Yeh directly given requirement se nikalta hai; hum kuch add nahi karte.
OPEN (tripped): sab traffic fallback (human reviewers / safe default) par jaata hai; koi model calls nahi hote. Ek cooldown timer run karta hai (ek fixed wait, jaise $60$ s ya $N$ events) jis dauraan breaker model ko bilkul touch karne se refuse karta hai — yeh rapid on/off flapping prevent karta hai.
HALF-OPEN (probing, automatically enter hota hai jab cooldown elapse ho jaata hai): breaker model ke through ek small trial batch let karta hai. Agar trial batch ka error rate below $t h r es h o l d$ wापas aa jaaye, toh CLOSED pe transition karo; agar abhi bhi high ho, toh OPEN wापas jaao aur cooldown restart karo.
RESET (explicit transition, HALF-OPEN → CLOSED): "reset" precisely woh successful HALF-OPEN probe hai — breaker apna error window clear karta hai aur normal CLOSED operation par return karta hai. Ek manual RESET bhi hai: ek on-call engineer, root cause fix karne ke baad, directly OPEN → CLOSED force kar sakta hai (window clear karte hue), automatic path ko override karte hue. Dono diagram mein labelled hain. Design note: sliding window recent degradation par react karta hai; startup rule early flapping prevent karta hai; cooldown oscillation prevent karta hai; HALF-OPEN probe (auto-reset) plus manual reset dono service restore karte hain; aur fallback guarantee karta hai ki users ko phir bhi koi answer mile. Yeh adversarial-robustness safety net in action hai.

Full state machine (CLOSED → OPEN → HALF-OPEN, plus dono reset paths) jo solution specify karta hai:

Exercise 4.2

Ek hospital diagnosis assistant ke liye ek monitoring dashboard design karo. Paanch panels ke liye, state karo woh failure mode jo har ek catch karta hai aur ek concrete alert trigger. Design ko teen pillars aur governance se tie karo.

Recall Solution

Neeche diya wireframe paanch panels ko ek annotated dashboard ki tarah sketch karta hai; accompanying reasoning har panel ke neeche state ki gayi hai.

Panel 1 — Human override rate. Catches: model ab useful nahi raha (doctors ise reject karte hain). Trigger: $6$ h ke liye $> 15%$ → page on-call. Panel 2 — Feature drift ( $D_{K L}$ ). Catches: inputs ab training world se match nahi karte. Trigger: $D_{K L} > 0.5$ → retrain queue karo. Panel 3 — Average confidence / entropy. Catches: model growing uncertain ho raha hai (out-of-distribution). Trigger: entropy baseline se $> 50%$ up. Panel 4 — Latency P95. Catches: usability degradation aur infra stress. Trigger: P95 $> 2$ s → auto-scale. Panel 5 — Demographic breakdown. Catches: emerging subgroup bias. Trigger: parity gap $> 5%$ → block + audit.

Yeh pillars se map kyun karta hai: override rate aur demographic breakdown human-oversight + fairness signals hain; drift, entropy, latency runtime monitoring hain; aur upar har threshold ek documented, auditable gate hai — exactly wahi jo ek governance framework aur MLOps discipline demand karte hain (har metric traceable, har alert owned).

Level 5 — Mastery

Exercise 5.1

Tumhara rollout budget ek stage mein at most $n = 5000$ samples observe karne ki permission deta hai, aur tum failure catch karne ka $α = 0.05$ confidence demand karte ho. Smallest failure rate $p$ kya hai jo tum is budget mein reliably detect kar sakte ho? Interpret karo: yeh us failures ke baare mein kya kehta hai jo isse rare hain?

Recall Solution

WHAT: rollout formula ko $n$ aur $α$ given hone par $p$ ke liye invert karo. $(1 - p)^{n} \leq α$ se starting: $n ln (1 - p) \leq ln α \Rightarrow ln (1 - p) \leq \frac{l n α}{n} \Rightarrow 1 - p \leq e^{l n α / n}$ $p \geq 1 - e^{l n (α) / n} = 1 - e^{l n (0.05) /5000} = 1 - e^{- 0.00059915}$ $p \geq 1 - 0.99940103 \approx 0.000599 = 0.0599%$ Interpretation: $5000$ samples ke saath tum roughly $0.06%$ tak failures catch kar sakte ho. Jo kuch bhi isse rare ho (jaise $0.01%$ , ten-thousand mein ek) woh is stage se $5%$ se zyada chance ke saath undetected nikal jaayega — yeh long-tail curse hai: tum ultra-rare events ke against testing se safety nahi pa sakte; unhe wild mein catch karne ke liye tumhe runtime monitoring aur circuit breakers chahiye.

Exercise 5.2

Ek confidence-based router predictions jo threshold $τ$ se neeche hain unhe humans ko bhejta hai. Maano $τ$ ko $0.85$ se $0.90$ tak badhane se human-review load $12%$ se $22%$ traffic tak badhta hai jabki automated errors $3.0%$ se $1.8%$ tak cut hote hain. Human review $\$ 0.50 $/ c a secos t k a r t ahai; ha r a u t o ma t e d er r or$ $40 $d o w n s t r e amha r m cos t k a r t ahai .$ 100{,}000$ daily requests ke liye, kaun sa threshold sasta hai? Kya cost sirf ek criterion hona chahiye?

Recall Solution

WHAT: total daily cost $=$ (human-review cost) $+$ (automated-error cost). $τ = 0.85$ par: review $= 0.12 \times 100000 \times \$ 0.50 = $6000 $. E r r or s$ = 0.030 \times 100000 \times $40 = $120000 $. T o t a l$ = $126000 $. * *$ \tau=0.90 $p a r : * * r e v i e w$ = 0.22 \times 100000 \times $0.50 = $11000 $. E r r or s$ = 0.018 \times 100000 \times $40 = $72000 $. T o t a l$ = $83000 $. * * S a s t a : * *$ \tau = 0.90 $,$ $43000 $/ d a y ba c ha t ahai . * * K y a cos t s i r f e k cr i t er i o nh o na c hahi y e ? N ahi . * * E k * m e d i c a l * se tt in g m e in, e k a u t o ma t e d er r or s i r f e k d o l l a r f i g u r e nahihai — i s k ama tl ab p a t i e n t ha r mh os ak t ahai, j o f u l l y m o n e t i z ab l e nahihai . E k [[6.4.14 - A I - g o v er nan ce f r am e w or k s ∣ g o v er nan ce p o l i cy]] cos t - o pt ima l se bhi z y a d ahi g h$ \tau$ (zyada human review) mandate kar sakti hai, aur explainability har routed case ke saath honi chahiye taki human reviewer acche se decide kar sake. Cost argument ka ek floor set karta hai, ceiling nahi.

Recall Self-Test Cloze

Ek staged rollout ko $n \geq \frac{l n α}{l n ( 1 - p )}$ samples chahiye kyunki $n$ tries mein zero failures ka chance == $(1 - p)^{n}$ == hai, $\leq α$ forced. Shadow mode A/B testing se isliye alag hai kyunki ::: sirf old model ka output users tak pahunchta hai; naya model silently offline score hota hai (zero user risk). Rising average entropy signal karta hai ::: model average pe kam confident ho raha hai, likely out-of-distribution inputs (drift) se mil raha hai. KL divergence undefined hai jab ek production bin empty ho kyunki ::: $lo g_{2} (P_{i} /0) = + \infty$ ; hum ise fix karne ke liye $Q$ ko Laplace-smooth karte hain. Ek circuit breaker OPEN se CLOSED aata hai ::: HALF-OPEN state (auto-reset probe) ya on-call engineer ke manual reset ke zariye.