Visual walkthrough — Mutation testing

This is a deep dive of Mutation testing. If a word or symbol is new, we define it the moment it appears — nothing is used before it is earned.

Step 1 — Start with one honest program

WHAT. We begin with a program that is correct and a small suite of tests that all pass.

def classify(n):
    if n > 0:          # the line we will attack
        return "pos"
    return "nonpos"

Tests (both currently pass):

assert classify(5)  == "pos"
assert classify(-3) == "nonpos"

WHY. Mutation testing never asks "is my code right?" — we assume the code is right. It asks a sharper question: "if my code were secretly wrong, would these tests notice?" To ask that, we first need a working baseline everyone agrees on. That green baseline is the picture below.

PICTURE. The number line (values of $n$) is split by a single fence at $n=0$. Everything right of the fence is "pos"; the fence and everything left is "nonpos". The two test inputs $n=5$ and $n=-3$ are the two dots — comfortably far from the fence.

Notice already: both test dots are far from the fence. Hold that thought — the whole story turns on it.

Step 2 — Inject one tiny bug: a mutant is born

WHAT. Apply the operator > → >= to our fence line. The program is now:

if n >= 0:   # mutant: the fence moved
    return "pos"

WHY this operator? Because > versus >= is the single most common real bug in the wild — the "off-by-one at the boundary." Mutation testing deliberately mimics the kinds of mistakes humans actually make, so a suite that kills these mutants is a suite that would have caught the real thing (this is why Boundary value analysis and mutation testing are cousins).

PICTURE. Watch the fence. In the original, $n=0$ was on the "nonpos" side. In the mutant, the fence slides so that $n=0$ flips to "pos". The only region that changed behaviour is the single point $n=0$, shaded below.

The changed zone is one dot wide. To catch this mutant, a test must land exactly on that dot.

Step 3 — Run the old tests: this mutant SURVIVES

WHAT. Feed our two existing tests to the mutant:

• classify(5) → still "pos" ✓ ($n=5$ is far right of the fence, unaffected)
• classify(-3) → still "nonpos" ✓ ($n=-3$ is far left, unaffected)

Both pass. Therefore the mutant survived. 😱

WHY it survived. Look back at Step 2's shaded zone: the only place the mutant misbehaves is $n=0$. Our test inputs are $n=5$ and $n=-3$ — neither of them is $0$. The tests literally never visit the one spot where the bug lives. A silent smoke alarm.

PICTURE. Overlay the "changed zone" (one dot at 0) with the "tested points" (dots at 5 and −3). They do not overlap. That non-overlap is the survival.

Step 4 — Not every survivor is your fault: the equivalent mutant

WHAT. Consider a different mutation on a different line. Suppose the code had a redundant guard:

if n > 0 and n > -1000000000:   # the second clause is always true for our domain

Mutating the second clause n > -1000000000 → n >= -1000000000 changes nothing observable: whenever the first clause n > 0 is true, $n$ is already far above $-10^9$, so both versions of the second clause are true. The mutant and original agree on every input.

WHY it matters. This mutant is unkillable — not because our tests are weak, but because there is nothing to catch. Punishing the suite for "failing" here would be blaming a firefighter for not extinguishing a fire that does not exist.

PICTURE. Two number lines, original and mutant, painted identically end to end — no shaded "changed zone" anywhere. Empty set of disagreement.

We must keep these separate from real survivors. That separation is the next step's job.

Step 5 — Sort every mutant into three bins (and meet the symbols)

WHAT. Generate all mutants (every operator on every eligible spot). Now we give each pile a name symbol as we introduce it:

• Let $M$ = the total number of mutants generated (all of them).
• Let $K$ = the number of killed mutants (some test failed).
• Let $S$ = the number of real survivors (tests all passed, yet a distinguishing input exists).
• Let $E$ = the number of equivalent mutants (unkillable by anyone).

Each mutant lands in exactly one of three bins after we run the suite and inspect it:

The three bins together account for every mutant, so $K+S+E=M$.

WHY three bins, not two? Because "survived" secretly contains two very different animals: real survivors $S$ (your tests are weak — fixable) and equivalent survivors $E$ (nothing to fix — a mirage). Mixing them would blame the suite for the mirage.

PICTURE. One big box of $M$ mutant tokens pours through a sorter into three coloured trays: killed $K$ (magenta), real survivors $S$ (orange), equivalent $E$ (violet).

Now, the crucial accounting move: how many mutants could ever be killed?

Step 6 — Count only what is catchable

WHAT. The equivalent mutants ($E$ of them) can never be killed by anyone. Remove them from the pool. What remains — the killable population — is: $\underset{\begin{array}{c}\text{all}\\ \text{mutants}\end{array}}{\underbrace{M}}-\underset{\begin{array}{c}\text{equivalent}\\ \text{(unkillable)}\end{array}}{\underbrace{E}}=\underset{\begin{array}{c}\text{killable}\\ \text{mutants}\end{array}}{\underbrace{M-E}}$

Reading it term by term: $M$ is everything we made; $E$ is the impossible-to-kill mirages; their difference $M-E$ is the honest denominator — the number of mutants that represent a real opportunity for a test to catch a bug. (Notice $M-E=K+S$: the killed plus the real survivors — exactly the catchable ones.)

WHY subtract instead of ignore? If we used $M$ as the denominator, a suite could never reach a perfect score just because a few equivalent mutants exist that nobody could ever kill. Subtracting $E$ makes the score measure the suite, not the mirages.

PICTURE. From Step 5's three trays, the violet (equivalent) tray is lifted out and set aside. Only the magenta + orange trays remain on the scale — that combined pile is $M-E$.

Step 7 — Assemble the score: successes over opportunities

WHAT. Divide killed ($K$) by killable ($M-E$).

WHY a ratio and not, say, just $K$? Because $K=40$ means nothing without knowing "out of how many?" Killing 40 of 48 catchable mutants ($\approx 83\%$) is strong; killing 40 of 4000 would be terrible. A ratio normalises to a comparable $0$-to-$1$ scale — a score of $1$ means every catchable bug is caught.

PICTURE. A horizontal bar of length $M-E$ (all opportunities). The killed part $K$ is filled magenta; the real survivors $M-E-K$ stay hollow orange. The fraction filled is the score.

Step 8 — The degenerate cases (so nothing surprises you)

Every case must be covered — here are the corners of the formula. Each corner gets its own WHY.

• All mutants killed, none equivalent ($K=M$, $E=0$): $\frac{M}{M-0}=1$. WHY. When every generated mutant is caught and there are no mirages to remove, numerator and denominator are the same count, so the fraction is exactly $1$ — the perfect suite, catching every injected bug.

• Every mutant is equivalent ($E=M$): denominator $M-E=0$, so the fraction is $\frac{0}{0}$ — undefined. WHY. If every mutant is a mirage, there was literally nothing killable to measure. Dividing by zero here is the maths politely saying "the question does not apply." Tools report "N/A", never $0$ — a $0$ would falsely accuse a fine suite.

• Empty test suite (no asserts at all): $K=0$, so score $=\frac{0}{M-E}=0$. WHY. With no assertions, no test can ever fail, so no mutant is killed — the numerator is stuck at $0$ while opportunities remain. This is the case that exposes Code coverage: empty tests can still execute every line for 100% coverage, yet catch nothing.

• No equivalent mutants ($E=0$): the formula collapses to the honest simple $K/M$. WHY. The whole point of the $-E$ term is to discount mirages. When there are none, there is nothing to discount, so the correction vanishes and the score is just "killed over total."

The one-picture summary

Everything above, compressed: mutants flow out of the code, sort into three trays, the equivalent tray is removed, and the score is the magenta slice of what remains.

Connections

• Mutation testing — this page derives its central formula from zero.
• Boundary value analysis — Step 2's > vs >= mutant is exactly a boundary bug.
• Code coverage — Step 8 shows why 100% coverage can still score $0$.
• Unit testing — the tests being graded here.
• Fault injection — the broader "break it on purpose and watch" family.