Worked examples — Fault tolerance — fail-safe vs fail-operational

This page is a drill sheet. The parent note gave you the formulas; here we hit every kind of input those formulas can face — the tiny, the huge, the "what if a number is zero", the trick question. Work each one before you read the steps.

Every example is broken into exactly three numbered steps (setup → core computation → finish), each with its own "Why this step?", so the rhythm never changes.

Two names you must not confuse

Before the matrix, pin down two acronyms this page uses side by side:

The scenario matrix

Before any numbers, let us list what can vary in a fault-tolerance problem, so we never meet a case we did not rehearse.

The two engines of this whole topic are:

Every symbol here was earned in the parent note. If any feels unfamiliar, pause and re-read it — we build on top now.

Here is the grid of case classes this topic can throw at you:

We now walk A → H, one example per cell. Related machinery lives in Redundancy patterns (N+1, TMR, DMR), Safety Integrity Levels (SIL), and Watchdog timers and health monitoring — link out when a step leans on them.

[!example] Example 1 — Cell A: near-perfect detection ($\beta \to 1$)

Statement. A pressure sensor fails at rate $\lambda =10^{-5}$ per hour. Diagnostics catch $\beta =0.99$ of failures and drive the valve to a safe (closed) state. Find MTTDf and say how many years that is.

Forecast: Guess first — will detecting 99% of failures buy you a little safety or a lot? Hold that number in your head.

1. Split the failure rate. The dangerous slice is ${\lambda }_d=(1-\beta )\lambda$. ${\lambda }_d=(1-0.99)(10^{-5})=0.01\times 10^{-5}=10^{-7}{\text{h}}^{-1}$ Why this step? Only undetected failures are dangerous — detected ones already steered you safe. So we throw away 99% of $\lambda$ before we even start.

2. Invert to get the mean time. MTTDf $=1/{\lambda }_d$. $\text{MTTDf}=\frac{1}{10^{-7}}=10^7\text{hours}$ Why this step? For a constant-rate process, mean-time-to-event is exactly the reciprocal of the rate (this is the ${\int }_0^{\infty }{e}^{-{\lambda }_{d}t}dt=1/{\lambda }_d$ from the parent note).

3. Convert to years. Divide by $8760$ hours/year. $10^7/8760\approx 1141\text{years}$ Why this step? A rate in "per hour" gives a time in hours; humans reason in years, so we rescale by the $8760$ hours in a year to make the "1141-year" safety claim legible. The unit conversion never changes the physics, only the readout.

Verify: Undetected fraction went from $100\%$ to $1\%$ — a $\times 100$ shrink of ${\lambda }_d$ — so MTTDf grew $\times 100$ over the "detect nothing" baseline of $10^5$ h. Units: $\frac{1}{{\text{h}}^{-1}}=\text{h}$. ✓ Detection is enormously powerful — see Ex 2 for the opposite extreme.

[!example] Example 2 — Cell B: the degenerate case $\beta =0$

Statement. Same sensor, $\lambda =10^{-5}$/h, but the diagnostics are disconnected: $\beta =0$. What is MTTDf now?

Forecast: With zero detection, does the formula blow up, break, or just give a plain number?

1. Plug $\beta =0$. ${\lambda }_d=(1-0)\lambda =\lambda =10^{-5}$. Why this step? Every failure is now dangerous — there is no safe branch at all. The system is no longer "fail-safe"; it is just a bare component, so MTTDf collapses onto plain MTTF.

2. Invert. $\text{MTTDf}=\frac{1}{10^{-5}}=10^5\text{hours}$ Why this step? Same reciprocal-of-rate rule as Ex 1 — but now the rate is the whole $\lambda$, because nothing was subtracted off.

3. Convert to years. $10^5/8760\approx 11.4\text{years}$. Why this step? The formula does not divide by zero — the danger $\lambda$ divided by $(1-\beta )$ only becomes infinite when $\beta \to 1$, not $\beta \to 0$. At $\beta =0$ MTTDf simply equals the raw MTTF $=1/\lambda$, a finite, legible number of years.

Verify: Compare to Ex 1: turning detection off drops MTTDf from $10^7$ to $10^5$ — exactly the factor $(1-\beta {)}^{-1}=100$ we gained before, now surrendered. ✓ The true blow-up ($\beta \to 1$) means "a system that never fails dangerously" — an idealisation, so the math correctly sends MTTDf to $\infty$ only there.

[!example] Example 3 — Cell C: realistic middling detection

Statement. An industrial actuator: $\lambda =4\times 10^{-6}$/h, and typical self-test coverage $\beta =0.6$. Find ${\lambda }_d$ and MTTDf in years.

Forecast: 60% coverage sounds decent — will it clear the "1000-year" bar from Ex 1?

1. Dangerous rate. ${\lambda }_d=(1-0.6)(4\times 10^{-6})=0.4\times 4\times 10^{-6}=1.6\times 10^{-6}$. Why this step? $40\%$ of failures slip through undetected, so we keep only that slice of $\lambda$.

2. MTTDf in hours. $\frac{1}{1.6\times 10^{-6}}=625,000\text{hours}$. Why this step? Reciprocal-of-rate again — the single move that turns any dangerous rate into a mean time.

3. Convert to years. $625000/8760\approx 71.3\text{years}$. Why this step? Rescale hours to years for a human-readable safety horizon, exactly as in Ex 1.

Verify: Only $71$ years — far below Ex 1's $1141$. Lesson: below $\beta \approx 0.9$ you are barely in the safe game; the Safety Integrity Levels (SIL) tables demand very high $\beta$ for exactly this reason. Units check: h. ✓

[!example] Example 4 — Cell D: TMR early life (redundancy's home turf)

Statement. Three identical flight computers, $\lambda =10^{-3}$/h. Evaluate at $\lambda t=0.1$ (early in the mission). Compare a single unit's survival to the TMR survival.

Forecast: Three copies instead of one — how much smaller does the failure chance get?

1. Single unit. $R_1=e^{-\lambda t}=e^{-0.1}\approx 0.9048$. Why this step? One component survives with the bare exponential — this is the chalk-blue baseline curve in the figure, our yardstick to beat.

2. TMR exact. Use the master formula with $\lambda t=0.1$: $R_{\text{sys}}=e^{-0.3}+3e^{-0.2}(1-e^{-0.1})$ $=0.7408+3(0.8187)(0.0952)\approx 0.7408+0.2338=0.9746$ Why this step? First term = all three alive; second term = exactly one dead but the two survivors still out-vote it. This is the chalk-pink curve sitting above the blue one at the dashed line — that height difference is what redundancy bought.

3. Read off the failure chances. Single fails $\approx 1-0.9048=9.5\%$; TMR fails $\approx 1-0.9746=2.5\%$. Why this step? "Reliability" is survival probability, so the failure chance is one minus it — the number a safety case actually cares about. We compute both so the win is stated in the language ("chance of losing the box") that engineers argue over.

Verify: $1-0.9048=0.0952$ and $1-0.9746=0.0254$; the ratio $0.0952/0.0254\approx 3.7\times$ improvement. ✓ This is the whole point of Redundancy patterns (N+1, TMR, DMR) — big wins where $\lambda t$ is small.

[!example] Example 5 — Cell E: the long-run limit (the surprise)

Statement. For the same TMR system, compute the MTTF (mean time to failure of the whole voted system) as a multiple of $1/\lambda$. Is it bigger or smaller than a single unit?

Forecast: Three components — surely the average lifetime is longer than one? Bet on it, then read.

1. Integrate the reliability. MTTF is the area under $R_{\text{sys}}(t)$; first expand the product into clean exponentials: $3e^{-2\lambda t}(1-e^{-\lambda t})=3e^{-2\lambda t}-3e^{-3\lambda t}$ ${\text{MTTF}}_{\text{TMR}}={\int }_0^{\infty }\left[e^{-3\lambda t}+3e^{-2\lambda t}-3e^{-3\lambda t}\right]dt$ Why this step? Mean lifetime = area under the survival curve (the same identity used for MTTDf in Ex 1). We expand first so every term is a pure exponential we can integrate with one rule.

2. Integrate term by term. Use ${\int }_0^{\infty }{e}^{-kt}dt=\frac{1}{k}$ for any positive constant $k$ (here $k=3\lambda ,2\lambda ,3\lambda$): $=\frac{1}{3\lambda }+\frac{3}{2\lambda }-\frac{3}{3\lambda }=\frac{1}{3\lambda }+\frac{3}{2\lambda }-\frac{1}{\lambda }$ Why this step? Each exponential decays to zero, so its total area is just the reciprocal of its decay rate $k$ — that single fact does all three integrals at once.

3. Combine over $6\lambda$. Put all three fractions on the common denominator $6\lambda$ and add: $=\frac{2}{6\lambda }+\frac{9}{6\lambda }-\frac{6}{6\lambda }=\frac{2+9-6}{6\lambda }=\frac{5}{6\lambda }\approx 0.833\frac{1}{\lambda }$ Why this step? The three pieces have denominators $3\lambda$, $2\lambda$, $\lambda$; $6\lambda$ is their least common multiple, so rewriting each over $6\lambda$ lets us add the numerators directly and collapse to a single clean fraction.

Verify: $\frac{5}{6}\approx 0.833<1$ — TMR's mean lifetime is shorter than a single unit's $1/\lambda$! Why? With no repair, TMR has more components that can accumulate the two failures needed to kill it, and the voter itself is unforgiving late in life. Redundancy helps early (Ex 4), not on average lifetime. See Ex 6 for the crossover point. ✓

[!example] Example 6 — Cell F: where does TMR stop helping?

Statement. Find the time $t_{\star }$ (as a multiple of $1/\lambda$) where TMR reliability equals a single unit's — beyond which redundancy is actually worse.

Forecast: Somewhere between "early" and "forever" the two curves must cross. Guess: before or after $t=1/\lambda$?

1. Set them equal with a substitution. Let $x=e^{-\lambda t}$ (so $x\in (0,1]$, decreasing in $t$), giving $x^3+3x^2(1-x)=x$ Why this step? Substituting $x$ turns the exponentials into a plain polynomial we can solve by hand — no calculator needed.

2. Reduce to a quadratic. Simplify $x^3+3x^2-3x^3=x\Rightarrow -2x^3+3x^2-x=0$, then divide by $x$: $2x^2-3x+1=0$ Why this step? Every term contains $x$, and $x=e^{-\lambda t}\ne 0$ for any finite time, so dividing by $x$ is legal and drops the cubic to a quadratic we recognise.

3. Solve and back-substitute. $x=\frac{3\pm \sqrt{9-8}}{4}=\frac{3\pm 1}{4}$ ⇒ $x=1$ or $x=\frac{1}{2}$. Discard $x=1$ (that is $t=0$, trivially equal); take $x=\frac{1}{2}$: $e^{-\lambda t_{\star }}=\frac{1}{2}\Rightarrow \lambda t_{\star }=\ln 2\approx 0.693\Rightarrow t_{\star }\approx \frac{0.693}{\lambda }$ Why this step? The quadratic gives both intersection heights; only $x=\frac{1}{2}$ is the meaningful crossover, and undoing the substitution with the natural log recovers the actual time.

Verify: Just before $t_{\star }$ (Ex 4 was at $\lambda t=0.1<0.693$) TMR led — consistent. In the figure the curves meet exactly at $\lambda t=\ln 2$, then blue overtakes pink. ✓ Real systems avoid the "worse" region by adding repair/replacement (Real-time scheduling with fault recovery) so they never sit past $t_{\star }$.

[!example] Example 7 — Cell G: pick the strategy (word problem)

Statement. You design the controller for a drawbridge over a shipping canal. If the controller faults, which strategy — fail-safe or fail-operational — and why?

Forecast: Think: what is the worst outcome of doing nothing vs doing something wrong?

1. Ask: is loss-of-function safe or dangerous? If the bridge simply stops moving and stays put, no ship is crushed and no car drives into a gap. Why this step? This is the single decision rule of the whole topic — before any redundancy math (Common-mode failures and diversity aside), you first classify whether doing nothing lands you in a safe state.

2. Compare the two failure costs. Doing nothing (bridge frozen) = traffic delay. Doing something wrong (bridge opens under a car / closes on a ship) = catastrophe. Why this step? Fail-safe is correct precisely when no-action is cheaper than wrong-action; here the two costs are wildly asymmetric, which decides the design.

3. Commit to fail-safe and spell out the mechanism. On any detected fault → clamp the bridge in its current position, drop the road barriers to RED, sound the alarm, and require a human reset before motion resumes. Why this step? We turn the judgement into hardware: spring-loaded brakes that engage on power loss (just like the railway signal relay in the parent note's Example 1) guarantee the safe state even if the software is dead, so no code defect can move the bridge unsafely. That physical default is what makes the choice trustworthy, not just the reasoning.

Verify: Contrast with an aircraft where freezing controls kills everyone — that one must be fail-operational. Same decision rule, opposite answer, because there "doing nothing" is the catastrophe. ✓ No arithmetic here, but the judgement is the exam's real target.

[!example] Example 8 — Cell H: the exam twist (counting done wrong)

Statement. A student writes: "TMR fails only if all 3 units fail, so $P_{\text{fail}}=p^3$." With per-unit failure probability $p=10^{-3}$ over the mission, find the correct $P_{\text{fail}}$ and how badly the student's shortcut misjudges the risk.

Forecast: Is "all three" the right count, or is it "two or more"? This is the classic slip — decide before reading on.

1. State the true failure condition. A majority voter loses when it can no longer form a majority — that happens at 2 or more failed units, not 3. $P_{\text{fail}}=\left(\genfrac{}{}{0ex}{}{3}{2}\right)p^2(1-p)+p^3$ Why this step? Two bad units already out-vote the one good one, so the system is dead the moment the second failure lands. Counting only "all 3 bad" ignores every "exactly 2 bad" arrangement and therefore undercounts the ways to fail.

2. Evaluate with $p=10^{-3}$. $=3(10^{-6})(0.999)+10^{-9}\approx 2.997\times 10^{-6}+10^{-9}\approx 3.0\times 10^{-6}$ Why this step? Plugging the number shows the "exactly 2" term ($\approx 3\times 10^{-6}$) dwarfs the "all 3" term ($10^{-9}$) by three orders of magnitude — the piece the student threw away is essentially the entire answer.

3. Compare to the student's shortcut. The student's $p^3=(10^{-3}{)}^3=10^{-9}$, so their estimate is $\frac{3.0\times 10^{-6}}{10^{-9}}\approx 3000\times \text{ too optimistic}.$ Why this step? Quantifying the error turns a subtle miscount into a safety-critical number: a design signed off on $10^{-9}$ that is really $3\times 10^{-6}$ has under-budgeted its risk by roughly 3000×.

Verify: Dominant term $3p^2=3\times 10^{-6}$ swamps $p^3=10^{-9}$, so ignoring the "exactly 2" case throws away 99.97% of the real risk. This matches the parent note's A320 figure ($\approx 3\times 10^{-6}$ per flight). ✓

[!recall]- Rapid self-test