Mutation testing: Question bank — Coding deep dive

Formula The three counts and the score

To read the edge-case answers you only need three letters:

$M$ = total number of mutants generated ("M" for Mutants).
$E$ = number of those that are Equivalent (unkillable).
$K$ = number Killed by the suite.

The killable population is $M - E$ (read: " $M$ minus $E$ ", i.e. total mutants with the unkillable ones removed). Why does $E$ leave the denominator rather than the numerator? The denominator counts opportunities to succeed, and an equivalent mutant is never a real opportunity (it cannot be killed by anyone), so it must be removed from that pool; putting it in the numerator would instead credit the suite for a "kill" that never happened. Hence: $Mutation Score = \frac{K}{M - E} (killed \div killable) .$

How to report it. The score is a fraction between $0$ and $1$ (e.g. $0.833$ ). To communicate it, multiply by $100$ and add a percent sign — $0.833 \to 83.3%$ . Both notations mean the same thing on this page; we write the fraction when doing arithmetic and the percentage when talking to humans.

True or false — justify

A test suite with 100% line coverage always kills every mutant.

False — coverage proves the line ran, not that any assertion would notice a wrong result; a test with no asserts hits 100% coverage and kills zero mutants.

A surviving mutant proves your production code has a bug.

False — the original code is fine; a survivor exposes a gap in the tests, telling you to add an assertion, not patch the code.

A killed mutant means your test suite is perfect.

False — it means that one injected bug was caught; other mutants may still survive, so one kill says nothing about the whole suite.

An equivalent mutant can eventually be killed if you write enough tests.

False — it behaves identically to the original for every possible input, so no test could ever distinguish them; killing it is logically impossible.

A mutation score of 100% is always the correct goal.

False — equivalent mutants often make 100% unreachable, and the last few percent cost enormous effort (diminishing returns); aim for high-value logic instead.

Two different mutation operators can produce the same mutant behaviour.

True — for example, on a variable already guaranteed positive (say a list length n >= 1), the mutations n > 0 → n >= 0 and n > 0 → True both behave identically to the original, because the differing input (n = 0) can never occur in that domain.

Mutation testing replaces unit testing.

False — it grades the unit tests you already wrote; with no tests there is nothing to run against the mutants, so it is a complement, not a replacement.

Adding one test can turn a survivor into a killed mutant.

True — a survivor means no current test covers the input where the mutant differs; the right new test (often a boundary value) closes that gap and kills it.

Every mutant that changes behaviour will be killed by a well-written suite.

True in principle — if a non-equivalent mutant differs on some input and the suite asserts on that input's output, it fails; surviving means that input was never exercised or never asserted.

Spot the error

"My mutation score is 40/50 = 80%, and 2 mutants are equivalent."

Wrong denominator — the 2 equivalent mutants can't be killed, so remove them from the killable set: score is

\frac{40}{50 - 2} = \frac{40}{48} \approx 0.833

(about 83.3%), written plainly as 40 killed over 48 killable. Using 50 in the denominator makes the fraction smaller, so it makes a good suite look worse than it is.

"This mutant survived, so I'll change the > back to >= in production."

You fixed the wrong file — the survivor points at the tests. The production code is correct; you must add a test (e.g. the boundary value) that would fail on the mutant.

"I got 100% coverage, so mutation testing would be a waste of time."

Coverage measures execution, not assertion strength. A suite can execute every line yet assert nothing meaningful, killing zero mutants — the exact gap mutation testing is built to catch.

"I'll count equivalent mutants as killed since they never break anything."

They are neither killed nor a suite weakness; they are removed from the killable set entirely (the

M - E

in the denominator, i.e. total minus equivalents). Marking them killed would falsely inflate the score.

"My suite kills every mutant on the happy path, so I'm done."

Happy-path kills leave boundary and error-path mutants untouched; survivors cluster at edges like n = 0 where > and >= diverge — those need explicit boundary tests.

"Mutation testing found no survivors on run 1, so I never need to run it again."

Every code change can introduce new untested logic; survivors reappear as the codebase grows, which is why teams run it in Continuous integration (often nightly).

Why questions

Why subtract equivalent mutants from the denominator instead of the numerator?

The logical step is: an equivalent mutant is never killable, therefore it must leave the killable set — that set is the denominator

M - E

(total mutants minus equivalents). Putting it in the numerator would instead reward the suite for catching an uncatchable thing, which is nonsense.

Why is a > → >= mutation such a common survivor?

The two operators differ on exactly one value (the boundary, e.g. n = 0); unless a test uses that value, both produce identical outputs everywhere else and the mutant hides.

Why does mutation testing usually run nightly rather than on every commit?

It reruns the whole suite once per mutant, so cost scales with (mutants × tests) — far slower than a single test run, making it impractical for every push.

Why is "code broke and nothing failed" the exact signal mutation testing wants?

That silence is the whole point of the technique: a broken program that passes all tests proves the tests aren't asserting on the broken behaviour — a measurable weakness.

Why can a test with 100% coverage still be worthless?

Coverage only records that lines executed; if no assertion checks the result, the test cannot fail on wrong output, so it detects no injected fault.

Why does Test-driven development tend to produce higher mutation scores?

TDD writes a failing test before the code, so every line exists to satisfy a specific assertion — meaning mutations to that line are far more likely to flip an asserted outcome.

Why is a high number of survivors near boundaries expected?

Many mutation operators target comparisons and off-by-one edges, which is precisely where Boundary value analysis says bugs and untested inputs concentrate.

Edge cases

A function has zero tests. What is its mutation score?

The score is exactly 0, not undefined: there are still

M - E

killable mutants, so the denominator is non-zero, and with no tests nothing can fail so

K = 0

, giving

\frac{0}{M - E} = 0

. The suite catches nothing.

No mutants are generated at all (so

M = 0

, and hence

E = 0

). What is the score?

Undefined — the denominator

M - E

0

, so the fraction has no value. This can happen for trivial code with nothing to mutate; treat it as "no information," not as a perfect score. It differs from the zero-tests case, where mutants did exist so the denominator stayed non-zero.

All generated mutants turn out equivalent (so

E

equals

M

). What is the score?

The killable set

M - E

0

, so the score is a division by zero and therefore undefined — there is nothing killable, meaning the tool tells you nothing about suite strength here. (Contrast the zero-tests case above, where the denominator was still non-zero.)

A mutant deletes a return statement in dead (unreachable) code. Killed or survived?

Survived, and likely equivalent — if the code can never run, its removal changes no observable behaviour, so no test can distinguish the versions.

A mutant changes True → False in a branch never taken by any test.

It survives, because the mutated branch is never exercised; it signals a missing test that reaches that branch, not a code fault.

max2(a,b): return a if a > b else b, test max2(3,1)==3. Mutant a > b → a >= b. Killed?

Survived — with 3 > 1 and 3 >= 1 both true, output is identical; you need an equal-input case like max2(5,5) to force the boundary and kill it.

Same function, mutant a > b → a < b, same test max2(3,1)==3. Killed?

Killed — 3 < 1 is false so the mutant returns b = 1 ≠ 3, the assertion fails, catching the injected bug.

A mutant makes code throw an exception on an input no test uses. Survived?

Survived — the crashing path is never triggered, so all tests pass; it flags an untested input, exactly the gap a new test should cover.

Your suite reaches score 1.0 but ships a real bug to production. Possible?

Yes — mutation operators only inject predefined small syntactic faults; a bug outside that catalogue (e.g. a wrong algorithm) is invisible to the score, so 1.0 is strong evidence, not a proof of correctness.

Connections

Unit testing — the tests this bank is quizzing you about.
Code coverage — the weaker metric several traps here contrast against.
Test-driven development — writing tests first tends to raise the score.
Boundary value analysis — where most survivors hide.
Continuous integration — where mutation runs, given its cost.
Fault injection — the broader "break it on purpose" family.