Exercises — Fault tolerance — fail-safe vs fail-operational

• (a) Fail-safe. If the flame sensor is unsure, cutting the gas valve is safer than pumping unlit gas. Loss of function (no heat) < incorrect function (gas leak/explosion).
• (b) Fail-operational. A pacemaker that shuts off = the heart may stop. It must keep pacing through faults, so it needs redundancy.
• (c) Fail-safe. E-stop de-energizes and clamps the belt. Stopped conveyor is safe; a runaway one is not.
• (d) Fail-operational. Losing flight control at cruise is catastrophic; the plane must keep flying with remaining computers.

Rule of thumb: ask "Is doing nothing safe?" If yes → fail-safe. If no → fail-operational.

${\lambda }_s$ = rate of safe failures (detected, sent to safe state). ${\lambda }_d$ = rate of dangerous failures (undetected, can cause harm). We want ${\lambda }_d$ small — dangerous failures are the only ones that hurt people. Total $\lambda$ is fixed by the hardware; the design goal is to shove as much of it as possible into the ${\lambda }_s$ bucket (raise $\beta$).

• DMR (Dual Modular Redundancy) → 2 units, cross-check (detect a disagreement, but can't tell which is wrong alone).
• TMR (Triple Modular Redundancy) → 3 units, majority vote (2 vs 1 identifies the liar).
• N+1 → one spare beyond the number actually needed.

See Redundancy patterns (N+1, TMR, DMR).

${\lambda }_d=(1-\beta )\lambda =0.05\times 2\times 10^{-5}=1\times 10^{-6}{\text{h}}^{-1}$. $\text{MTTDf}=1/{\lambda }_d=1/10^{-6}=1\times 10^6\text{h}$. In years: $10^6/8760\approx 114.2\text{yr}$. Why the integral formula works: $\text{MTTDf}={\int }_0^{\infty }{e}^{-{\lambda }_{d}t}dt=1/{\lambda }_d$. The area under the survival curve is the mean lifetime — a wide, slow-decaying curve (small ${\lambda }_d$) has huge area = long life.

$\lambda t=10^{-4}\times 5=5\times 10^{-4}$, so $p=1-e^{-5\times 10^{-4}}\approx 4.99875\times 10^{-4}$. "At least 2 of 3 work" = NOT ("2 or 3 failed"). System fails only if $\ge 2$ fail: $P(\text{fail})=\left(\genfrac{}{}{0ex}{}{3}{2}\right)p^2(1-p)+p^3\approx 3p^2\approx 3\times (5\times 10^{-4}{)}^2=7.5\times 10^{-7}.$ $R_{\text{sys}}=1-P(\text{fail})\approx 1-7.5\times 10^{-7}\approx 0.99999925$. Why drop $p^3$? It is $\sim 10^{-10}$ — a millionth of the $p^2$ term. Keeping it changes no decimal that matters.

Worst case: fault occurs just after a kick, so up to $20\text{ms}$ passes before the first miss; then $2$ missed kicks to declare = $2\times 20=40\text{ms}$... but the standard count is from fault to declaration. Detection latency $\le 2\times 20=40\text{ms}$, then actuation $8\text{ms}$: total $\le 40+8=48\text{ms}$. $48\text{ms}\le 50\text{ms}$ → meets the deadline (with $2\text{ms}$ margin). See Watchdog timers and health monitoring. Why margin matters: with only $2\text{ms}$ slack, any clock jitter or an extra missed kick blows the budget. A safe design targets margin, not a bare pass.

Set $R_{\text{sys}}=R_1$: $3e^{-2\lambda t}-2e^{-3\lambda t}=e^{-\lambda t}$. Let $u=e^{-\lambda t}$ (so $u\in (0,1]$): $3u^2-2u^3=u\Rightarrow 3u-2u^2=1\Rightarrow 2u^2-3u+1=0\Rightarrow (2u-1)(u-1)=0.$ Roots $u=1$ ($t=0$, trivial) and $u=\frac{1}{2}$. So the meaningful crossover is $e^{-\lambda t}=\frac{1}{2}\Rightarrow \lambda t=\ln 2\approx 0.693$. Read the figure: for $\lambda t<\ln 2$ (early life, left of the crossing) TMR is above the single line — it's more reliable. For $\lambda t>\ln 2$ the extra components (3× the parts, 3× the failure exposure) drag it below. TMR buys early-life reliability, not longevity.

${\int }_0^{\infty }3e^{-2\lambda t}dt=\frac{3}{2\lambda }$ and ${\int }_0^{\infty }2e^{-3\lambda t}dt=\frac{2}{3\lambda }$. ${\text{MTTF}}_{\text{TMR}}=\frac{3}{2\lambda }-\frac{2}{3\lambda }=\frac{9-4}{6\lambda }=\frac{5}{6\lambda }\approx \frac{0.833}{\lambda }.$ Single unit: $1/\lambda$. So TMR's average life is shorter ($0.833/\lambda <1/\lambda$). Why: without repair, TMR degrades to a 2-of-3 that then can't survive the next failure; after the second failure it's dead. It trades away the long, thin tail (where a lone unit occasionally survives a long time) for a fat, safe early survival hump. Average-of-lifetime hides the whole point — early reliability is what safety cares about.

Before: ${\lambda }_{d,1}=(1-0.60)\lambda =0.40\lambda$. After: ${\lambda }_{d,2}=(1-0.99)\lambda =0.01\lambda$. $\text{MTTDf}\propto 1/{\lambda }_d$, so improvement factor $=\frac{{\lambda }_{d,1}}{{\lambda }_{d,2}}=\frac{0.40}{0.01}=40$. Interpretation: a 40× longer time between dangerous failures — without touching the hardware's raw quality, purely by making failures self-revealing. This is exactly why Safety Integrity Levels (SIL) reward high diagnostic coverage.

• Fail-operational (must keep working long enough to stop safely): steering + braking actuation, ego-localization, the "come to a controlled stop" trajectory planner. Use TMR or DMR+diverse-monitor so one compute fault doesn't drop control mid-manoeuvre. This is the minimal safe stop capability — it must survive one fault.
• Fail-safe (graceful shutdown is acceptable once stopped): the full autonomous driving stack (perception, routing). If it faults, degrade to the fail-operational "safe stop" lane, brake to a halt, hazard lights on.
• Redundancy vs diversity: use redundancy (identical units) for random hardware faults; use diversity (different sensors/algorithms — camera + LiDAR + radar; two independently-written planners) to beat common-mode software bugs that would kill all identical copies at once.
• Architecture in one line: fail-operational down to the minimal "controlled stop," then fail-safe by executing that stop. This "fail-degraded-then-safe" pattern is the industry norm for L4 vehicles. See Real-time scheduling with fault recovery for making the safe-stop task meet its deadline.

Single channel PFD $=5\times 10^{-2}$ — far short of $10^{-3}$. Use 1oo2 (two independent channels, either can trip): a dangerous failure needs both to fail dangerously. If independent, ${\text{PFD}}_{sys}\approx {\text{PFD}}_1\times {\text{PFD}}_2=(5\times 10^{-2}{)}^2=2.5\times 10^{-3}$. Still above $10^{-3}$. Add a third independent channel or improve each channel's PFD. E.g. three independent channels: $(5\times 10^{-2}{)}^3=1.25\times 10^{-4}<10^{-3}$ ✓. Caveat / the real engineering: you cannot assume perfect independence — common-mode failures put a floor under the product (the β-factor model). So use diversity across channels, not clones. See Safety Integrity Levels (SIL).

Identical: $\text{PFD}=q^2+c=(2\times 10^{-2}{)}^2+10^{-3}=4\times 10^{-4}+1\times 10^{-3}=1.4\times 10^{-3}$. Note the common-mode term dominates ($10^{-3}\gg 4\times 10^{-4}$) — the redundancy is being wasted because both copies share the same flaw. Diverse: ${\text{PFD}}^{\prime }=q^2+c^{\prime }=4\times 10^{-4}+5\times 10^{-5}=4.5\times 10^{-4}$. Improvement factor $=\frac{1.4\times 10^{-3}}{4.5\times 10^{-4}}\approx 3.11$. Lesson: once you add redundancy, the residual risk is set by whatever the copies share. Diversity attacks that shared term — see Common-mode failures and diversity and Byzantine fault tolerance for faults where channels actively disagree.

The voter sits in series with the voted core: everything must survive AND the voter must survive. $R_{\text{TMR}+v}(t)=R_v(t)\cdot (3e^{-2\lambda t}-2e^{-3\lambda t})=e^{-{\lambda }_{v}t}(3e^{-2\lambda t}-2e^{-3\lambda t}).$ Because it's a series element, its failures add linearly ($\approx {\lambda }_{v}t$) while the voted core's failures are quadratic ($\approx 3(\lambda t{)}^2$). For the voter not to dominate we need its linear term below the core's quadratic term: ${\lambda }_{v}t\ll 3(\lambda t{)}^2⟹{\lambda }_v\ll 3{\lambda }^{2}t.$ The deep point: a cheap, un-redundant voter can silently become the system's single point of failure — the very thing TMR was meant to remove. This is why safety-critical voters are themselves made simple, formally verified (see Formal verification of safety properties), or replicated. Verify the whole chain with Hardware safety mechanisms and Byzantine fault tolerance in mind.

Ask "is doing nothing safe?" ::: yes → fail-safe; no → fail-operational ${\lambda }_d=?$ ::: $(1-\beta )\lambda$, the dangerous (undetected) failure rate When does TMR beat a single unit? ::: for $\lambda t<\ln 2\approx 0.693$ (early life) ${\text{MTTF}}_{\text{TMR}}=?$ ::: $\frac{5}{6\lambda }\approx 0.833/\lambda$ — less than a single unit's $1/\lambda$ What sets residual risk after adding redundancy? ::: the common-mode term the copies share — attack it with diversity Why can a voter ruin TMR? ::: it's a series element; its linear failure rate can dominate the core's quadratic one