5.5.26 · D1Embedded Systems & Real-Time Software

Foundations — Fault tolerance — fail-safe vs fail-operational

1,938 words9 min readBack to topic

This page assumes you have seen nothing. Every letter, every squiggle, every fraction in the parent note the parent topic is built here from the ground up, in the order you need them. Never skip ahead — each symbol is earned before it is used.


What is a "failure rate" and why a letter for it?

WHY a rate and not just "chance of failing"? Because failure builds up over time. A part that survives one hour easily might not survive a thousand hours. A rate lets us ask about any length of time. Look at the picture below.

Figure — Fault tolerance — fail-safe vs fail-operational

The red staircase counts failures as time marches right. Its average steepness is exactly — that is the whole meaning of the symbol.


The number and "surviving to time "

We keep seeing in the parent note. Let us earn every piece of it.

Figure — Fault tolerance — fail-safe vs fail-operational

The letter — probability, a number between 0 and 1

So is just the same idea written two ways. And the chance it has failed by time is everything left over:


Splitting failures: , and the fraction

Not every failure is equally bad. A signal that goes dark by accident (annoying but safe) is different from one that shows GREEN when it should be RED (deadly). The parent note splits into two streams.

Now the total rate splits into two smaller rates, like a river forking:

Figure — Fault tolerance — fail-safe vs fail-operational

WHY split it? Because fail-safe design only cares about the dangerous trickle . Making close to squeezes that trickle almost to nothing — that is the entire engineering game of fail-safe systems.


The integral and "mean time until..."

The parent note writes . Two new symbols: and MTTDf.

Apply the same machine to the dangerous stream and you get the parent's headline result:


Counting redundant units: and voting

Fail-operational systems use spares. To count how many ways spares can fail, we need one more notation.

Figure — Fault tolerance — fail-safe vs fail-operational

This is why the parent writes the TMR reliability as a sum: "all three good" plus "exactly one bad" — the two situations where the majority still wins.


The remaining alphabet, quickly earned


How the foundations feed the topic

failure rate lambda

reliability R equals e to minus lambda t

the number e

elapsed time t

probability P from 0 to 1

MTTF equals 1 over lambda

integral adds tiny slices

safe fraction beta

split into lambda safe and lambda dangerous

Fail-safe MTTDf

n choose k counting

TMR voting reliability

Fail-operational availability A

Fault tolerance topic


Equipment checklist

Say the answer out loud before revealing. If any stump you, re-read that section.

What does measure, and in what units?
The failure rate — average failures per hour, units .
What does represent?
The reliability : probability one part is still working after hours.
Why is the exponent negative?
Survival only decreases; a positive exponent would mean parts un-break.
What is in terms of ?
, because alive + failed must total 1.
What does mean, and what is ?
is the fraction of failures that are safe/detected; is the dangerous failure rate.
What does the integral compute, and what does it equal?
The area under the reliability curve = the mean time to failure = .
What does equal and why does it appear in TMR?
It equals 3 — the number of ways exactly one of three units can be the failed one.
When does a TMR system actually fail?
Only when 2 or more of the 3 units fail, since the majority voter overrides a single bad unit.
What does measure, and what is "four nines"?
Availability — probability the system is operational at time ; four nines means .
What are and ?
is the deadline to reach the safe state; is a tiny accepted failure probability (e.g. ).