Visual walkthrough — Fault tolerance — fail-safe vs fail-operational

5.5.26 · D2 · Coding › Embedded Systems & Real-Time Software › Fault tolerance — fail-safe vs fail-operational

Is page par ek hi promise hai parent note se — "redundancy ek system ko tab bhi chalata rehta hai jab koi part toot jaaye" — aur hum is maths ko bilkul scratch se rebuild karenge. End tak tumne woh curve khud draw kar li hogi jo ek aircraft engineer ko batati hai: mere triple-redundant flight computer ke 10 ghante baad bhi zinda rehne ki kitni probability hai?

Hum har symbol pehle build karte hain, phir use karte hain. Agar tumne pehle kabhi $e$, probability, ya exponent nahi dekha, toh line one se shuru karo aur summary tak pahuncho.

Jo prerequisites kaam aayenge: Redundancy patterns (N+1, TMR, DMR), aur yeh parent topic ki child note hai.

---

Step 1 — "Reliability" ka matlab kya hai? Ek component ko ek aisi coin ki tarah draw karo jo time ke saath flip hoti rehti hai

KYA. Hum ek single component se shuru karte hain — ek flight-control computer. Hum ek number invent karte hain jise reliability kehte hain, likha jaata hai $R(t)$, jo sirf ek sawaal ka jawaab deta hai:

"Agar main ise time $0$ par on karta hoon, toh time $t$ par bhi kaam karne ki kitni chance hai?"

YEH number kyun, koi aur kyun nahi. Engineers yeh promise nahi kar sakte ki "yeh kabhi nahi tootega." Sab kuch aakhirkar toot jaata hai. Toh yes/no promise ki jagah, hum ek probability use karte hain — $0$ (impossible) aur $1$ (certain) ke beech ka ek number. $R(t)=1$ matlab pakka zinda; $R(t)=0$ matlab pakka dead. Hum dekhte hain ki time ke saath yeh number 1 se 0 ki taraf kaise slide karta hai.

PICTURE. Horizontal track dekho: time left se right flow karta hai. Cyan bar woh chance hai ki component abhi bhi zinda hai. $t=0$ par yeh poora bar fill karta hai (brand new, $R=1$). Jaise time beetha, bar shrink hoti hai — amber slice woh chance hai ki woh pehle hi mar chuka hai.

---

Step 2 — "Hazard rate" $\lambda$: bar kitni tezi se shrink hoti hai

KYA. Hume batana hoga ki alive-bar kitni tezi se shrink ho rahi hai. Hum ek symbol introduce karte hain, ==$\lambda$ (Greek letter "lambda"), hazard rate== — failures per hour.

$\lambda =\underset{\text{ek chhota number}}{\underbrace{10^{-4}}}\underset{\text{kitne}}{\underbrace{\text{failures}}}/\underset{\text{is itne time mein}}{\underbrace{\text{hour}}}$

Padhne ka tarika: har ek ghante mein, $10^{-4}$ (das hazaar mein se ek) chance hai ki yeh component us ghante mein mar jaaye.

Ek constant kyun. Electronics ke normal life mein (infant-mortality ke baad, wear-out se pehle), "is ghante" marne ki chance age par depend nahi karti — ek 3-ghante-purana chip ek 1-ghante-purane chip se zyada thaka nahi hota. Yahi memoryless property hai jo $\lambda$ ko ek fixed number rehne deti hai. Dekho Watchdog timers and health monitoring ki hum field mein failures actually kaise measure karte hain.

PICTURE. Har ghante hum jo bhi alive-bar bachi hai usse same fraction $\lambda$ kaat lete hain. Ek fixed amount nahi — ek fixed fraction. Steps dekho: har drop current bar ka ek slice hai, isliye drops physically chhoti hoti jaati hain jaise bar shrink hoti hai.

---

Step 3 — $e^{-\lambda t}$ kyun aata hai (aur $e$ yahan kya kar raha hai)

KYA. Same fraction ko baar baar, aur baar baar barikhi se kaatne par exactly ek hi curve banti hai:

$R(t)=e^{-\lambda t}$

Hum har symbol earn karte hain.

$R(t)=\underset{\stackrel{\begin{array}{c}\text{kitna decay}\\ \text{pile up hua hai}\end{array}}{\overbrace{-\lambda t}}}{\underbrace{e}}$

• $\lambda t$ = rate $\times$ time = time $t$ tak accumulated decay ki total amount. Agar $\lambda =10^{-4}$/h aur $t=10$ h, toh $\lambda t=10^{-3}$.
• Minus sign kehta hai decay bar ko shrink karta hai.
• $e$ (Euler's number, $\approx 2.718$) woh natural base hai jo tab bhi aata hai jab koi cheez apne current size ke ek fixed proportion se change hoti hai. Yahi Step 2 ki picture hai exactly.

$e$ kyun, say, $\frac{1}{2}$ kyun nahi? Kyunki decay continuous hai — coin har ghante ek baar flip nahi hoti, yeh "flip" har pal ho rahi hai. Agar tum "fraction $\lambda$ lose karo" ko ever-smaller time slices mein apply karo aur slices ko zero tak jaane do, toh limit $e^{-\lambda t}$ hai. Yeh repeated fractional loss ka smooth version hai.

PICTURE. Step 2 ki staircase, smoothly ek curve mein convert hui. Height $1$ se shuru hoti hai, shuru mein sabse tezi se girти hai, aur zero ko hug karti hai (lekin kabhi touch nahi karti). Amber dot hamare example ko mark karta hai: $\lambda t=10^{-3}$ par, height $\approx 0.999$ hai.

---

Step 4 — Teen computers: sab tarike draw karo jinse woh ji ya mar sakte hain

KYA. Triple Modular Redundancy (TMR) teen identical units aur ek voter use karta hai — ek chhota circuit jo teeno answers leta hai aur jo majority kehti hai woh output karta hai. System "alive" tab tak hai jab tak kam se kam 2 of 3 units kaam kar rahi hain (voter ko do agreeing votes chahiye).

$p=1-e^{-\lambda t}$ ek unit ke time $t$ par dead hone ki chance hai, aur $q=e^{-\lambda t}$ iske alive hone ki chance (note $p+q=1$).

2-of-3 kyun. Ek akeli healthy unit par akela trust nahi kiya ja sakta — agar yeh ek tooti unit se disagree kare, toh voter nahi bata sakta kaun sahi hai. Do agreeing units ek liar ko out-vote kar dete hain. Toh humein $\ge 2$ alive chahiye.

PICTURE. 3 units ke har possible outcome ka ek tree — har branch alive (cyan) ya dead (amber) hai. Kul aath leaves. Framed leaves (all-3-alive, aur teeno 2-alive cases) woh hain jahan system survive karta hai.

---

Step 5 — Binomial se survivor branches count karo

KYA. Hum framed branches ko jod lete hain. Yeh choose karna ki kaun si units survivors hain — kitne tarike hain — $\left(\genfrac{}{}{0ex}{}{3}{k}\right)$ ("3 choose $k$") se likha jaata hai — kitne distinct leaves mein exactly $k$ alive hain.

$\left(\genfrac{}{}{0ex}{}{3}{3}\right)=1\text{(ek tarika: sab alive)}\left(\genfrac{}{}{0ex}{}{3}{2}\right)=3\text{(teen tarike: choose karo kaun mara)}$

System survive karta hai agar 3 alive ya exactly 2 alive:

$R_{\text{TMR}}(t)=\underset{\text{sab 3 alive}}{\underbrace{\left(\genfrac{}{}{0ex}{}{3}{3}\right)q^3}}+\underset{\begin{array}{c}\text{exactly 2 alive,}\\ \text{1 dead}\end{array}}{\underbrace{\left(\genfrac{}{}{0ex}{}{3}{2}\right)q^{2}p}}$

$q=e^{-\lambda t}$ aur $p=1-e^{-\lambda t}$ substitute karo:

$R_{\text{TMR}}(t)=\underset{\text{teeno alive}}{\underbrace{e^{-3\lambda t}}}+\underset{\text{ek specific unit dead, baaki do alive, }\times 3}{\underbrace{3e^{-2\lambda t}(1-e^{-\lambda t})}}$

Probabilities multiply kyun karte hain. Teen units independently fail hote hain (alag chips, alag power lines — dekho Common-mode failures and diversity jab yeh assumption toot jaaye). Independent events ki joint probability product hoti hai: $q\cdot q\cdot q=q^3$.

PICTURE. Do stacked reliability curves: single unit (thin cyan) aur TMR (thick cyan). Notice karo TMR pehle single curve ke upar rehti hai, phir zyada right mein neeche chali jaati hai.

---

Step 6 — Early-life win: real numbers plug in karo

KYA. Aircraft ko flight ki parwah hoti hai, jo component ki life mein early hoti hai ($\lambda t$ chhota). $\lambda t=0.1$ lo (yani $t=0.1/\lambda$):

$R_{\text{single}}=e^{-0.1}=0.905$

$R_{\text{TMR}}=e^{-0.3}+3e^{-0.2}(1-e^{-0.1})=0.7408+3(0.8187)(0.0952)=0.9746$

TMR ke liye term by term:

• $e^{-0.3}=0.7408$ → teeno ke survive karne ki chance.
• $3\times e^{-0.2}\times (1-e^{-0.1})=3\times 0.8187\times 0.0952=0.2338$ → exactly ek ke marne ki chance.
• Sum $=0.9746$.

Failure chance $9.5\%$ (single) se $2.5\%$ (TMR) tak gir jaati hai — jahan zaroori hai wahan lagbhag 4× safer.

Yeh yahan kyun kaam karta hai lekin baad mein nahi. Pehle, ek doosri failure rare hoti hai (dono ko marna hoga), isliye pehli failure ko tolerate karna almost free pass hai.

PICTURE. Step 5 ke plot ka left mein zoom. $R=1$ top line tak do amber gaps: single-unit gap bada hai ($9.5\%$), TMR gap chhota hai ($2.5\%$).

---

Step 7 — Surprising long-run twist (degenerate case)

KYA. Mean Time To Failure (average lifetime) paane ke liye $R_{\text{TMR}}$ ko sab time par integrate karo:

${\text{MTTF}}_{\text{TMR}}={\int }_0^{\infty }{R}_{\text{TMR}}(t)dt=\frac{1}{3\lambda }\cdot 1+3\left(\frac{1}{2\lambda }-\frac{1}{3\lambda }\right)=\frac{5}{6\lambda }$

Lekin ek single unit ka MTTF $\frac{1}{\lambda }$ hai. Toh:

${\text{MTTF}}_{\text{TMR}}=\frac{5}{6\lambda }\approx 0.83\times \frac{1}{\lambda }<{\text{MTTF}}_{\text{single}}$

Teen units ki average life ek se CHHOTI hoti hai!

Yeh contradiction kyun nahi hai. Saare time par average karo toh TMR haarta hai — kyunki bahut baad mein, 2-of-3 survive karna 1-of-1 se mushkil hota hai (pehle se hi zyada cheezein toot chuki hain, aur koi repair nahi). Lekin hum infinity tak fly nahi karte. Finite mission window par jo matter karta hai, TMR kaafi safer hai (Step 6). Yeh Step 5 ki do curves ka crossing point hai.

PICTURE. Do curves cross ho rahi hain. Crossing ke left (short missions): TMR upar = safer. Uske right (long unattended life): single upar. Amber vertical line crossover mark karta hai.

---

Ek-picture summary

Is final drawing mein saaton steps compress hain: ek shrinking bar ($e^{-\lambda t}$) se, 2-of-3 branch count tak, do crossing curves aur early-life safety gain tak.

Recall Feynman retelling — apne plain words mein bolo

Ek single computer ke abhi bhi zinda rehne ki chance 1 se shuru hoti hai aur ek smooth curve $e^{-\lambda t}$ se neeche slide karti hai, jahan $\lambda$ hai kitne failures per hour hote hain aur $\lambda t$ hai kitna "decay" pile up hua hai. Curve $e$-shaped hai kyunki cheez har pal jo bacha hai uska same fraction lose karti hai — bilkul ek savings account ki tarah jo fixed percent bleed kare.

Ab teen identical computers ek voter ke peeche rakho jo majority par trust karta hai. System tab tak jeeta hai jab tak do of teen zinda hain. Main teen coins ke lande ka poora aath-way tree list karta hoon, chaar ko rakhta hoon jahan kam se kam do alive hain, aur unki probabilities add karta hoon — multiply karke kyunki units independently fail hote hain. Is se milta hai $R_{\text{TMR}}=e^{-3\lambda t}+3e^{-2\lambda t}(1-e^{-\lambda t})$.

Life mein pehle yeh triple system kaafi safer hai: $\lambda t=0.1$ par failure chance $9.5\%$ se $2.5\%$ tak gir jaati hai. Lekin saare time par integrate karo aur TMR ki average lifetime sirf $\frac{5}{6\lambda }$ hai — single unit ke $\frac{1}{\lambda }$ se chhoti! Koi paradox nahi: redundancy long-run average life ko short-run safety ke liye trade karti hai. 10-hour flight par — jahan mission per safety hi sab kuch hai — exactly yahi trade hum chahte hain.

---

Related deep prerequisites & extensions: Byzantine fault tolerance (jab ek faulty unit die hone ki jagah jhooth bolta hai), Real-time scheduling with fault recovery, Hardware safety mechanisms, aur Formal verification of safety properties.