Fault tolerance — fail-safe vs fail-operational

5.5.26 · Coding › Embedded Systems & Real-Time Software

Overview

Fault tolerance ek system ki woh ability hai jisme components fail hone ke bawajood system sahi se kaam karta rehta hai. Do fundamental strategies define karti hain ki systems faults ke response mein kya karte hain: fail-safe (nuksan rokne ke liye shut down karo) aur fail-operational (failures ke bawajood kaam karte raho).

[!intuition] Core Intuition

Ek elevator aur ek airplane ke baare mein socho:

• Elevator (fail-safe): Agar cable sensors koi problem detect karein, toh emergency brakes TURANT clamp ho jaati hain. Elevator ruk jaata hai. Passengers phanse rehte hain lekin safe hote hain. System "kuch galat karo" se zyada "kuch mat karo" ko prefer karta hai.
• Airplane (fail-operational): Agar ek engine fail ho jaaye, toh plane ko baaki engine(s) se ZAROOR uda rehna chahiye. Beech mein shut down karna catastrophic hoga. System ko component loss ke bawajood critical functions maintain karne hote hain.

Humein alag alag strategies ki ZAROORAT kyun hai? Kyunki cost of failure alag alag hoti hai:

• Fail-safe: Function ka kho jaana galat function se ZYADA SAFE hai (nuclear reactors, railway signals, medical devices)
• Fail-operational: Function ka kho jaana degraded function se ZYADA KHATARNAK hai (aircraft, spacecraft, pacemakers, self-driving cars highway speeds par)

[!definition] Formal Definitions

Fail-Safe

Ek aisa system jo fault detect hone par safe state mein transition karta hai jahan woh nuksan nahi pahuncha sakta. Safe state typically yeh hoti hai:

• Passive safe: System power down ho jaata hai, lock out ho jaata hai, ya physically disconnect ho jaata hai (e.g., E-stop button)
• Active safe: System actively safety maintain karta hai (e.g., brakes engaged rakhe rehna)

Mathematical model: Agar time $t_f$ par fault detect hua, toh system ko deadline $\Delta t_{max}$ ke andar safe state $S_{safe}$ mein pahunchna chahiye: $P(\text{reach }{S}_{safe}|t<t_f+\Delta t_{max})\ge 1-ϵ$ jahan $ϵ$ acceptably small hai (e.g., SIL 4 safety integrity level ke liye $10^{-9}$).

Fail-Operational

Ek aisa system jo faults ke bawajood critical functions perform karta rehta hai, usually redundancy ke saath. Key properties:

• Graceful degradation: Non-critical features kho sakti hain, lekin safety-critical functions bani rehti hain
• No single point of failure: Multiple components fail ho sakte hain bina total loss ke

Availability requirement: Mission time $T$ aur fault arrival rate $\lambda$ ke liye: $A(T)=P(\text{system operational at }T)\ge A_{target}$ Typically critical systems ke liye $A_{target}>0.9999$ (four nines) hoti hai.

[!formula] First Principles se Reliability Derive Karna

Fail-Safe: Mean Time to Dangerous Failure (MTTDf)

Shuru karo: Ek component constant hazard rate $\lambda$ se fail hota hai. Hum probability compute karna chahte hain ki ek failure danger ki taraf le jaayegi.

Step 1: Safe failure fraction $\beta$ define karo. Jab failure hoti hai:

• Probability $\beta$: failure detect ho jaati hai aur system safe state mein jaata hai
• Probability $(1-\beta )$: failure undetected rehti hai (dangerous)

Step 2: Total failure rate $\lambda$ split hoti hai: $\lambda ={\lambda }_s+{\lambda }_d$ jahan ${\lambda }_s=\beta \lambda$ (safe failures), ${\lambda }_d=(1-\beta )\lambda$ (dangerous).

Step 3: Time $t$ par dangerous failure probability: $P_{\text{dangerous}}(t)=1-e^{-{\lambda }_{d}t}$

Step 4: Mean Time to Dangerous Failure: $\text{MTTDf}={\int }_0^{\infty }{e}^{-{\lambda }_{d}t}dt=\frac{1}{{\lambda }_d}=\frac{1}{(1-\beta )\lambda }$

Yeh kyun matter karta hai: $\lambda =10^{-5}$ failures/hour aur $\beta =0.99$ (99% safe failures) wale safety system ke liye: ${\lambda }_d=(1-\beta )\lambda =0.01\times 10^{-5}=10^{-7}{\text{h}}^{-1}$ $\text{MTTDf}=\frac{1}{{\lambda }_d}=\frac{1}{10^{-7}}=10^7\text{hours}\approx 1141\text{years}$ (kyunki $10^7\text{h}÷8760\text{h/yr}\approx 1141\text{yr}$). Failures ko self-revealing banana (high $\beta$) safety mein teen orders of magnitude ka faayda deta hai.

Fail-Operational: Redundancy aur MTBF

Scenario: Triple Modular Redundancy (TMR) — majority voter ke saath teen identical units.

Step 1: Single unit ka failure rate $\lambda$ hai, MTTF = $1/\lambda$.

Step 2: System tabhi fail hota hai jab 2+ units fail ho jaayein. Binomial use karo:

• Probability ki teeno time $t$ par kaam kar rahe hain: $[e^{-\lambda t}{]}^3=e^{-3\lambda t}$
• Probability ki exactly 2 kaam kar rahe hain (1 fail hua): $\left(\genfrac{}{}{0ex}{}{3}{1}\right)e^{-2\lambda t}(1-e^{-\lambda t})$

Step 3: System operational hai agar 2+ units kaam kar rahi hain: $R_{\text{sys}}(t)=e^{-3\lambda t}+3e^{-2\lambda t}(1-e^{-\lambda t})$

Step 4: Chote $\lambda t$ (early life) ke liye, expand karo $(1-e^{-\lambda t})\approx \lambda t$: $R_{\text{sys}}(t)\approx e^{-3\lambda t}+3\lambda t{e}^{-2\lambda t}$

Step 5: TMR ke liye MTTF ($R_{\text{sys}}$ ko 0 se $\infty$ tak integrate karo): ${\text{MTTF}}_{\text{TMR}}=\frac{5}{6\lambda }$

Yeh surprising kyun hai: 3x components ke saath, MTTF sirf single component ka ~0.83x hai! Lekin: Early-life reliability bahut behtar hoti hai. $t=0.1/\lambda$ par (toh $\lambda t=0.1$):

• Single: $R(0.1/\lambda )=e^{-0.1}\approx 0.905$
• TMR (exact): $R_{\text{sys}}=e^{-0.3}+3e^{-0.2}(1-e^{-0.1})\approx 0.7408+3(0.8187)(0.0952)\approx 0.9746$
• TMR (first-order approx): $e^{-0.3}+3(0.1)e^{-0.2}\approx 0.7408+0.2456\approx 0.9864$

Toh TMR is early-life point par failure probability ko ~9.5% se ~2.5% tak reduce karta hai — jahan zyada matter karta hai wahan ek bada faayda.

[!example] Example 1: Railway Signal System (Fail-Safe)

Scenario: Track signal train entry control karta hai. Agar uncertain ho toh RED dikhana ZAROORI hai.

Design:

1. Lamp filament: Continuously monitor kiya jaata hai. Agar current drop ho (burnt filament), system failure detect karta hai.
2. Fail-safe relay: Spring-loaded, de-energize hone ke liye. GREEN dikhane ke liye continuous power chahiye hoti hai.
3. Fault response: Koi bhi sensor fault → relay de-energize → signal defaults to RED.

Har step WHY?

• Filament monitoring: Detect karta hai ki signal physically red/green dikhane mein capable hai ya nahi
• Spring-loaded relay: Power loss par safe state (RED) guarantee karta hai — koi software nahi chahiye
• Default RED: Trains rukenge. Delay hoga, accident nahi.

Code check (simplified):

enum State { GREEN, RED };
State signal_state = RED;  // Default safe
 
void update_signal() {
    if (filament_ok() && track_clear() && power_ok()) {
        signal_state = GREEN;
    } else {
        signal_state = RED;  // ANY fault → safe state
    }
}

Key property: Code defect signal ko GREEN nahi rakh sakta jab unsafe ho — hardware relay kisi bhi anomaly par RED enforce karta hai.

[!example] Example 2: Aircraft Flight Control (Fail-Operational)

Scenario: Fly-by-wire Airbus A320 — pilot stick inputs computers ke through control surfaces tak jaate hain. Cruise altitude par computer failures tolerate KARNI CHAHIYE.

Design:

1. Triple redundancy: 3 identical flight control computers (FCCs), har ek same software run karta hai.
2. Dual redundancy: 2 elevator computers (ELACs), 3 spoiler/aileron computers (SECs).
3. Voting: Har control surface multiple computers se commands receive karta hai. Physical actuator average ya vote karta hai.
4. Dissimilar redundancy: Kuch backup alag hardware + software implementation use karta hai.

Har step WHY?

• 3 FCCs: Agar 1 fail ho, majority vote (2 vs 1) faulty wale ko identify aur override karta hai.
• Multiple surface controllers: Agar ELAC-1 fail ho, ELAC-2 turant elevator control le leta hai.
• Dissimilar backup: Common-mode failures se protect karta hai (e.g., software bug jo saare identical units ko affect kare).

Graceful degradation path:

• Normal: Saare systems → envelope protection ke saath full fly-by-wire (Normal Law)
• Kuch computers lost: Alternate Law — reduced protections, phir bhi full software fly-by-wire
• Aur zyada loss: Direct Law — stick directly control surfaces se map hota hai bina envelope protection ke, lekin ABHI BHI software-driven fly-by-wire (mechanical cable-and-lever system nahi). Aircraft controllable rehta hai.
• A320 par bahut limited mechanical backup sirf rudder + trimmable horizontal stabilizer hai, jo purely aircraft ko steady rakhne ke liye use hota hai jab computers/power restore ho rahi ho — full manual flight nahi.

Math check: 3 FCCs ke saath, $\lambda =10^{-4}$ failures/hour, 10-hour flight: $P(\text{≥2 fail})=P(\text{exactly 2})+P(\text{all 3})=\left(\genfrac{}{}{0ex}{}{3}{2}\right)p^2(1-p)+p^3$ jahan $p=1-e^{-10^{-4}\cdot 10}\approx 10^{-3}$: $P(\text{≥2 fail})\approx 3\times (10^{-3}{)}^2+(10^{-3}{)}^3\approx 3\times 10^{-6}$

~333,000 mein se ek flight voting kho degi. Backup ELACs/SECs aur degraded control laws ke saath, actual risk orders of magnitude kam hai.

[!example] Example 3: Insulin Pump (Hybrid Approach)

Scenario: Implanted insulin pump ko precise doses deliver karni hoti hain. Overdose fatal hai; underdose hyperglycemia cause karta hai.

Design:

1. Fail-operational chote faults ke liye: Dual processors cross-check karte hain. Agar ek glitch kare, doosra le leta hai.
2. Fail-safe persistent faults ke liye: Agar error >3 checks tak bani rahe, pump delivery band kar deta hai aur alarm bajata hai.
3. Watchdog timer: Independent hardware timer. Pump ko har 100ms mein watchdog ko "pet" karna hota hai. Miss → shutdown.

Hybrid WHY?

• Transient faults (cosmic ray bit flip): Fail-operational. Jaldi resume karo.
• Persistent faults (processor damage): Fail-safe. Nuksan pahunchane se pehle roko.

Verification:

• Har dose calculation par dual-processor compare. Mismatch → halt.
• Watchdog ensure karta hai ki software crash nahi hua hai (infinite loop). Hardware safe state force karta hai.

[!mistake] Common Mistakes

1: "Redundancy hamesha safety improve karti hai"

Yeh sahi kyun lagta hai: Zyada backups = zyada reliability, hai na?

Asli baat: Redundancy nayi failure modes introduce kar sakti hai:

• Complexity: Zyada components = zyada interaction failures. Byzantine faults jahan ek unit conflicting outputs deta hai.
• Common-mode failures: Saare redundant units same bug, power supply, ya environmental stress share karte hain.
• Voting logic errors: Agar voter khud hi faulty hai, toh redundancy bekar hai.

Example: Boeing 737 MAX MCAS TWO angle-of-attack sensors use karta tha lekin ek waqt mein SIRF EK read karta tha per flight (alternating flights). Ek bad sensor ne crashes ka karan bana kyunki system dono ko simultaneously compare nahi karta tha.

Fix:

• Dissimilar redundancy use karo (alag hardware/software).
• Voters ko simpler aur independently verified banao.
• Common-mode failures ke liye test karo (thermal, radiation, power transients).

Mistake 2: "Fail-safe ka matlab hai 'band kar do'"

Yeh sahi kyun lagta hai: Safe state = no power = off.

Asli baat: Kuch systems ko active safe states chahiye hoti hain:

• Holding brakes: Brakes LAGANE hote hain (active), release nahi karne hote.
• Containment: Nuclear reactor control rods ko fault par INSERT karna hota hai (active).
• Ventilation: Biolab ko faults ke dauran bhi negative pressure (active) maintain karni hoti hai.

Fix: Safe state application-specific hoti hai. Poochho: "Kaunsi state nuksan rokegi agar saara control kho jaaye?" Phir hardware defaults + fail-safe actuators design karo taaki woh state enforce ho sake.

Mistake 3: "Fail-operational hamesha gracefully degrade kar sakta hai"

Yeh sahi kyun lagta hai: Ek subsystem kho do, baaki uthaa lete hain.

Asli baat: Kuch systems mein cliff failures hote hain:

• Threshold effects: 2 engines wala aircraft ud sakta hai. 0 engines wala aircraft indefinitely glide nahi kar sakta.
• Cascading failures: Ek overloaded redundant unit fail hota hai, doosron par load badhta hai, chain collapse hoti hai.

Example: Power grid blackouts. Ek line kho do → padosi load uthaa lete hain → padosi overload ho jaate hain → cascade.

Fix:

• Spare capacity ke liye design karo (N+2 redundancy, sirf N+1 nahi).
• Load shedding implement karo (catastrophic failure se pehle non-critical functions drop karo).
• Cascading failure paths model karo.

[!recall]- Ek 12-Saal ke Bachche ko Explain Karo

Socho tumhare paas ek bahut important toy robot hai jise hamesha kaam karte rehna hai.

Fail-safe aise hai jaise ek bada laal STOP button ho. Agar kuch galat ho jaaye (jaise koi wire loose ho jaaye), robot turant freeze ho jaata hai aur band ho jaata hai. Woh kuch bhi dangerous nahi karega. Yeh un cheezoon ke liye perfect hai jaise ek robot jo pagal ho jaaye toh kisi ko hurt kar sakta hai — galat direction mein chalta rehne se better hai ruk jaana.

Fail-operational aise hai jaise robot mein DO brains hon. Agar ek brain toot jaaye, doosra brain robot ko chalata rehta hai! Robot thoda slower chal sakta hai ya naach nahi sakta, lekin woh chalata rehta hai aur girta nahi. Yeh un cheezoon ke liye perfect hai jinhein KAAM KARTE REHNA CHAHIYE, jaise Mars par ek robot (ruk nahi sakte — wahan koi fix karne nahi hai!).

Apni bicycle ke baare mein socho:

• Brakes = fail-safe: Agar brake cable toot jaaye, brake pads wheel ko touch karna band kar dete hain. Tum brake nahi kar sakte, lekin wheel atkegi nahi (jo tumhe crash kara deta).
• Do wheels = fail-operational: Agar puncture ho jaaye, tum doosre wheel par (slowly, carefully) chal sakte ho stranded hone ki bajaye.

Bada idea: Kuch cheezein RUKNE mein zyada safe hoti hain jab tooti hoon (fail-safe). Doosri cheezein CHALTE REHNE mein zyada safe hoti hain tooti hone par bhi (fail-operational). Engineers choose karte hain is basis par ki kya zyada dangerous hai!

[!mnemonic] Memory Aids

FAIL-SAFE = "Stop And Freeze Everything"

• Stop immediately
• Alarm/Alert
• Freeze in known-good state
• Ensure no harm

FAIL-OPERATIONAL = "Keep Running Despite Damage"

• Keep critical functions alive
• Redundancy required
• Degrade gracefully

Kaun sa kab use karein?

• "RUKNA safe hai" → Fail-safe (elevators, e-stops, traffic lights jo default red hoon)
• "RUKNA danger hai" → Fail-operational (planes, spacecraft, life support)

Connections

• Watchdog timers and health monitoring — Fail-safe systems faults kaise detect karte hain
• Redundancy patterns (N+1, TMR, DMR) — Fail-operational systems ke architectures
• Safety Integrity Levels (SIL) — Required MTTDf quantify karna
• Common-mode failures and diversity — Redundancy kyun kaafi nahi hoti
• Byzantine fault tolerance — Malicious ya inconsistent failures handle karna
• Real-time scheduling with fault recovery — Failover ke baad timing guarantees
• Hardware safety mechanisms — Physical fail-safe relays, watchdogs
• Formal verification of safety properties — Fail-safe transitions prove karna

Flashcards

Concept Map