Fault tolerance — fail-safe vs fail-operational: Exercises

5.5.26 · D4 · Coding › Embedded Systems & Real-Time Software › Fault tolerance — fail-safe vs fail-operational

Yeh page ek graded ladder hai. Har problem ko Solution kholne se pehle khud try karo. Yahan use hone wale har symbol ki definition parent note mein hai — agar kuch samajh na aaye, pehle parent topic revisit karo.

Quick symbol refresher (taaki koi undefined term na mile):

Definition Is page par use hone wale Symbols

$λ$ (lambda) — failure rate: average mein ek ghante mein kitni failures hoti hain. Units: $h^{- 1}$ (per hour). Bada $λ$ = zyada baar fail hota hai.
$R (t)$ — reliability: probability ki cheez time $t$ par abhi bhi kaam kar rahi hai. $1$ se shuru hota hai (brand new) aur $0$ ki taraf decay karta hai.
$e^{- λ t}$ — survival curve. Isko aise padho: "time $t$ ke baad ab tak kuch fail nahi hua, uski chance." Minus sign matlab yeh shrink karta hai jaise time badhta hai.
$β$ (beta) — safe failure fraction: saari failures mein se wo fraction jo pakdi jaati hain aur system ko safe state mein le jaati hain.
MTTF / MTTDf — Mean Time To Failure / Mean Time To Dangerous Failure: (kisi bhi / ek dangerous) failure se pehle average kitne ghante. Jitna zyada = utna safe.
$A$ — availability: kitne fraction time system up aur usable hai.

Level 1 — Recognition

L1.1 — Kaunsi strategy?

Har system ke liye fail-safe ya fail-operational batao, aur ek-line reason do: (a) Ek gas boiler ka flame sensor. (b) Ek pacemaker. (c) Ek factory conveyor ka E-stop. (d) Ek commercial jet ka fly-by-wire.

Recall Solution

(a) Fail-safe. Agar flame sensor unsure hai, toh gas valve band karna unlit gas pump karte rehne se zyada safe hai. Function ka loss (no heat) < incorrect function (gas leak/explosion).
(b) Fail-operational. Jo pacemaker shut off ho jaaye = heart ruk sakta hai. Isko faults ke through pacing karte rehna hoga, isliye redundancy chahiye.
(c) Fail-safe. E-stop de-energize karta hai aur belt clamp kar deta hai. Ruka hua conveyor safe hai; runaway conveyor nahi.
(d) Fail-operational. Cruise par flight control kho dena catastrophic hai; plane ko baaki computers ke saath uda rehna hoga.

Rule of thumb: poochho "Kya kuch na karna safe hai?" Agar haan → fail-safe. Agar nahi → fail-operational.

L1.2 — Equation padho

Parent $λ = λ_{s} + λ_{d}$ deta hai. Seedhe alfaaz mein, $λ_{s}$ aur $λ_{d}$ kya hain, aur hum kaun sa chhota chahte hain?

Recall Solution

$λ_{s}$ = safe failures ki rate (detect ho gayi, safe state mein bhej di). $λ_{d}$ = dangerous failures ki rate (undetected, harm cause kar sakti hai). Hum $λ_{d}$ chhota chahte hain — dangerous failures hi wo hain jo logon ko hurt karti hain. Total $λ$ hardware se fix hota hai; design goal hai jitna ho sake zyada $λ_{s}$ bucket mein daalo (yaani $β$ badhao).

L1.3 — Pattern ko count se match karo

Match karo: N+1, DMR, TMR ko "2 units, cross-check", "3 units, majority vote", "zaroorat se ek spare zyada" se.

Recall Solution

DMR (Dual Modular Redundancy) → 2 units, cross-check (ek disagreement detect karo, lekin akele nahi bata sakte kaun galat hai).
TMR (Triple Modular Redundancy) → 3 units, majority vote (2 vs 1 jhootay ko identify karta hai).
N+1 → actually zaroorat se ek zyada spare.

Dekho Redundancy patterns (N+1, TMR, DMR).

Level 2 — Application

L2.1 — Smoke detector ka MTTDf

Ek detector ka $λ = 2 \times 1 0^{- 5} h^{- 1}$ aur safe failure fraction $β = 0.95$ hai. $λ_{d}$ aur Mean Time To Dangerous Failure hours aur years mein nikalo ( $1 yr = 8760 h$ ).

Recall Solution

$λ_{d} = (1 - β) λ = 0.05 \times 2 \times 1 0^{- 5} = 1 \times 1 0^{- 6} h^{- 1}$ . $MTTDf = 1/ λ_{d} = 1/1 0^{- 6} = 1 \times 1 0^{6} h$ . Years mein: $1 0^{6} /8760 \approx 114.2 yr$ . Kyun integral formula kaam karta hai: $MTTDf = \int_{0}^{\infty} e^{- λ_{d} t} d t = 1/ λ_{d}$ . Survival curve ke neeche ka area hi mean lifetime hai — ek wide, slowly decay hoti curve (chhota $λ_{d}$ ) ka area bahut bada hota hai = lambi life.

L2.2 — Ek specific time par two-out-of-three reliability

Teen FCCs, har ek ka $λ = 1 0^{- 4} h^{- 1}$ , flight time $t = 5 h$ . $p = 1 - e^{- λ t}$ use karke (probability ki ek unit fail ho gayi hai), $p$ aur $R_{sys} = P (at least 2 of 3 work)$ calculate karo.

Recall Solution

$λ t = 1 0^{- 4} \times 5 = 5 \times 1 0^{- 4}$ , toh $p = 1 - e^{- 5 \times 1 0^{- 4}} \approx 4.99875 \times 1 0^{- 4}$ . "Kam se kam 2 of 3 kaam kar rahe hain" = NOT ("2 ya 3 fail"). System tabhi fail hota hai jab $\geq 2$ fail hoon: $P (fail) = (2 3) p^{2} (1 - p) + p^{3} \approx 3 p^{2} \approx 3 \times (5 \times 1 0^{- 4})^{2} = 7.5 \times 1 0^{- 7} .$ $R_{sys} = 1 - P (fail) \approx 1 - 7.5 \times 1 0^{- 7} \approx 0.99999925$ . $p^{3}$ kyun drop karein? Yeh $\sim 1 0^{- 10}$ hai — $p^{2}$ term ka ek millionth. Isko rakhne se koi meaningful decimal nahi badlega.

L2.3 — Deadline check

Ek fail-safe pressure controller ko $Δ t_{ma x} = 50 ms$ ke andar apna safe state reach karna hai. Watchdog har $20 ms$ pe fire karta hai, aur fault declare karne ke liye zyada se zyada $2$ missed kicks lagte hain, plus vent valve kholne mein $8 ms$ lagte hain. Kya yeh deadline meet karta hai?

Recall Solution

Worst case: fault ek kick ke just baad hota hai, toh pehli miss se pehle $20 ms$ tak guzar sakta hai; phir declare karne ke liye $2$ missed kicks = $2 \times 20 = 40 ms$ ... lekin standard count fault se declaration tak hai. Detection latency $\leq 2 \times 20 = 40 ms$ , phir actuation $8 ms$ : total $\leq 40 + 8 = 48 ms$ . $48 ms \leq 50 ms$ → deadline meet karta hai ( $2 ms$ margin ke saath). Dekho Watchdog timers and health monitoring. Kyun margin matter karta hai: sirf $2 ms$ slack ke saath, koi bhi clock jitter ya ek extra missed kick budget udaa dega. Safe design margin target karta hai, sirf barely pass nahi.

Level 3 — Analysis

L3.1 — TMR single unit se kab behtar hai?

Single-unit reliability $R_{1} (t) = e^{- λ t}$ hai. TMR $R_{sys} (t) = 3 e^{- 2 λ t} - 2 e^{- 3 λ t}$ hai. $x = λ t$ ki wo value nikalo jahan dono curves cross karti hain (uske neeche TMR jeet jaata hai). Figure dekho.

Figure — Fault tolerance — fail-safe vs fail-operational

Recall Solution

$R_{sys} = R_{1}$ set karo: $3 e^{- 2 λ t} - 2 e^{- 3 λ t} = e^{- λ t}$ . $u = e^{- λ t}$ lo (toh $u \in (0, 1]$ ): $3 u^{2} - 2 u^{3} = u \Rightarrow 3 u - 2 u^{2} = 1 \Rightarrow 2 u^{2} - 3 u + 1 = 0 \Rightarrow (2 u - 1) (u - 1) = 0.$ Roots $u = 1$ ( $t = 0$ , trivial) aur $u = \frac{1}{2}$ . Toh meaningful crossover hai $e^{- λ t} = \frac{1}{2} \Rightarrow λ t = ln 2 \approx 0.693$ . Figure padho: $λ t < ln 2$ ke liye (early life, crossing ke left) TMR single line se upar hai — yeh zyada reliable hai. $λ t > ln 2$ ke liye extra components (3× parts, 3× failure exposure) isko neeche kheench dete hain. TMR early-life reliability deta hai, longevity nahi.

L3.2 — TMR ka MTTF sirf $\frac{5}{6 λ}$ kyun hai

$R_{sys} (t) = 3 e^{- 2 λ t} - 2 e^{- 3 λ t}$ ko $0$ se $\infty$ tak integrate karo $MTTF_{TMR} = \frac{5}{6 λ}$ confirm karne ke liye, aur explain karo ki yeh single-unit $1/ λ$ se kam kyun hai.

Recall Solution

$\int_{0}^{\infty} 3 e^{- 2 λ t} d t = \frac{3}{2 λ}$ aur $\int_{0}^{\infty} 2 e^{- 3 λ t} d t = \frac{2}{3 λ}$ . $MTTF_{TMR} = \frac{3}{2 λ} - \frac{2}{3 λ} = \frac{9 - 4}{6 λ} = \frac{5}{6 λ} \approx \frac{0.833}{λ} .$ Single unit: $1/ λ$ . Toh TMR ki average life chhoti hai ( $0.833/ λ < 1/ λ$ ). Kyun: repair ke bina, TMR ek aisi 2-of-3 mein degrade ho jaata hai jo agla failure survive nahi kar sakti; doosri failure ke baad yeh dead hai. Yeh wo long, thin tail (jahan ek akela unit kabhi kabhi lamba survive karta hai) ko trade kar deta hai ek fat, safe early survival hump ke liye. Life-ka-average saari baat chhupa deta hai — safety jis cheez ki parwah karti hai wo hai early reliability.

L3.3 — Diagnostic coverage $β$ ko lift karta hai

Ek controller ka raw $λ = 4 \times 1 0^{- 6} h^{- 1}$ hai. Self-test add karne se detection itni badhti hai ki $β$ , $0.60$ se $0.99$ ho jaata hai. MTTDf kitne factor se improve hota hai?

Recall Solution

Pehle: $λ_{d, 1} = (1 - 0.60) λ = 0.40 λ$ . Baad mein: $λ_{d, 2} = (1 - 0.99) λ = 0.01 λ$ . $MTTDf \propto 1/ λ_{d}$ , toh improvement factor $= \frac{λ _{d, 1}}{λ _{d, 2}} = \frac{0.40}{0.01} = 40$ . Interpretation: dangerous failures ke beech 40× zyada lamba time — hardware ki raw quality ko touch kiye bina, sirf failures ko self-revealing banane se. Yahi reason hai ki Safety Integrity Levels (SIL) high diagnostic coverage ko reward karte hain.

Level 4 — Synthesis

L4.1 — Hybrid strategy design karo

Ek autonomous shuttle pedestrian zone mein $40 km/h$ pe chalta hai. Ek fault-tolerance architecture sketch karo: kaun se functions fail-operational hain, kaun se fail-safe hain, aur redundancy vs. diversity kahaan lagaoge.

Recall Solution

Fail-operational (safely rokne ke liye itna der tak kaam karta rehna chahiye): steering + braking actuation, ego-localization, "controlled stop tak aao" trajectory planner. Ek compute fault mid-manoeuvre control drop na kare isliye TMR ya DMR+diverse-monitor use karo. Yeh minimal safe stop capability hai — isko ek fault survive karna hoga.
Fail-safe (ruk jaane ke baad graceful shutdown acceptable hai): poora autonomous driving stack (perception, routing). Agar fault ho, fail-operational "safe stop" lane mein degrade karo, halt tak brake karo, hazard lights on karo.
Redundancy vs diversity: random hardware faults ke liye redundancy (identical units) use karo; common-mode software bugs jo saari identical copies ko ek saath kill kar dete hain unhe beat karne ke liye diversity (alag sensors/algorithms — camera + LiDAR + radar; do independently-likhe planners) use karo.
Architecture ek line mein: minimal "controlled stop" tak fail-operational, phir us stop ko execute karke fail-safe. Yeh "fail-degraded-then-safe" pattern L4 vehicles ke liye industry norm hai. Safe-stop task ko uski deadline meet karaane ke liye dekho Real-time scheduling with fault recovery.

L4.2 — SIL choose karo

Ek machine ka hazard serious injury cause karta hai; exposure ~constant hai; escape mushkil hai; demand rate low hai (din mein kuch operations). Risk graph SIL 3 point karta hai, jiske liye $PFD_{a v g} < 1 0^{- 3}$ chahiye (average probability of failure on demand). Tumhara single channel $PFD = 5 \times 1 0^{- 2}$ deta hai. Ek fix propose karo aur new PFD estimate karo (assume karo ki independent channels multiply karte hain).

Recall Solution

Single channel PFD $= 5 \times 1 0^{- 2}$ — $1 0^{- 3}$ se bahut door. Use 1oo2 (do independent channels, koi bhi trip kar sakta hai): ek dangerous failure ke liye dono ko dangerously fail karna hoga. Agar independent hain, $PFD_{sy s} \approx PFD_{1} \times PFD_{2} = (5 \times 1 0^{- 2})^{2} = 2.5 \times 1 0^{- 3}$ . Abhi bhi $1 0^{- 3}$ se upar. Ek teesra independent channel add karo ya har channel ka PFD improve karo. Jaise teen independent channels: $(5 \times 1 0^{- 2})^{3} = 1.25 \times 1 0^{- 4} < 1 0^{- 3}$ ✓. Caveat / real engineering: tum perfect independence assume nahi kar sakte — common-mode failures product ke neeche ek floor laga dete hain (the β-factor model). Toh channels mein clones ki jagah diversity use karo. Dekho Safety Integrity Levels (SIL).

Level 5 — Mastery

L5.1 — Common-mode ke under diverse pair vs identical pair

Do channels mein se har ek ka random-fault PFD $q = 2 \times 1 0^{- 2}$ hai. Ek common-mode fault (shared bug/environment) probability $c = 1 \times 1 0^{- 3}$ ke saath dono ko ek saath hit karta hai, random faults se independent. System dangerous-failure probability ko $PFD_{sy s} = q^{2} + c$ model karo (dono random-fail hoon, YA common-mode). Identical pair ke liye compute karo. Phir ek diverse pair $c$ ko $c^{'} = 5 \times 1 0^{- 5}$ tak reduce karta hai (alag code, alag sensor). Diverse PFD aur improvement factor compute karo.

Recall Solution

Identical: $PFD = q^{2} + c = (2 \times 1 0^{- 2})^{2} + 1 0^{- 3} = 4 \times 1 0^{- 4} + 1 \times 1 0^{- 3} = 1.4 \times 1 0^{- 3}$ . Note karo ki common-mode term dominate karta hai ( $1 0^{- 3} ≫ 4 \times 1 0^{- 4}$ ) — redundancy waste ho rahi hai kyunki dono copies same flaw share karti hain. Diverse: $PFD^{'} = q^{2} + c^{'} = 4 \times 1 0^{- 4} + 5 \times 1 0^{- 5} = 4.5 \times 1 0^{- 4}$ . Improvement factor $= \frac{1.4 \times 1 0 ^{- 3}}{4.5 \times 1 0 ^{- 4}} \approx 3.11$ . Lesson: ek baar redundancy add karo, residual risk wo cheez set karta hai jo copies share karti hain. Diversity us shared term par attack karta hai — dekho Common-mode failures and diversity aur Byzantine fault tolerance un faults ke liye jahan channels actively disagree karte hain.

L5.2 — Voter as a single point of failure

TMR ek voter par rely karta hai. Suppose har channel ki reliability $R = e^{- λ t}$ hai aur voter ki reliability $R_{v} = e^{- λ_{v} t}$ hai. Asli TMR-with-voter reliability likho, aur $λ_{v}$ par wo condition batao jis par voter failure budget dominate na kare.

Recall Solution

Voter voted core ke saath series mein hai: sab kuch survive karna chahiye AUR voter survive karna chahiye. $R_{TMR + v} (t) = R_{v} (t) \cdot (3 e^{- 2 λ t} - 2 e^{- 3 λ t}) = e^{- λ_{v} t} (3 e^{- 2 λ t} - 2 e^{- 3 λ t}) .$ Kyunki yeh ek series element hai, iske failures linearly add hote hain ( $\approx λ_{v} t$ ) jabki voted core ki failures quadratic hain ( $\approx 3 (λ t)^{2}$ ). Voter dominate na kare isliye uska linear term core ke quadratic term se neeche chahiye: $λ_{v} t ≪ 3 (λ t)^{2} ⟹ λ_{v} ≪ 3 λ^{2} t .$ Deep point: ek sasta, un-redundant voter silently system ka single point of failure ban sakta hai — bilkul wahi cheez jo TMR remove karne ke liye tha. Isliye safety-critical voters khud simple, formally verified (dekho Formal verification of safety properties), ya replicated banaye jaate hain. Poori chain verify karo Hardware safety mechanisms aur Byzantine fault tolerance ko dhyan mein rakh ke.

Recall Self-test checklist

Poochho "kya kuch na karna safe hai?" ::: haan → fail-safe; nahi → fail-operational $λ_{d} = ?$ ::: $(1 - β) λ$ , dangerous (undetected) failure rate TMR single unit se kab better hai? ::: $λ t < ln 2 \approx 0.693$ ke liye (early life) $MTTF_{TMR} = ?$ ::: $\frac{5}{6 λ} \approx 0.833/ λ$ — single unit ke $1/ λ$ se kam Redundancy add karne ke baad residual risk kya set karta hai? ::: common-mode term jo copies share karti hain — ise diversity se attack karo Voter TMR kyun kharab kar sakta hai? ::: yeh ek series element hai; iska linear failure rate core ke quadratic wale ko dominate kar sakta hai