Mutation testing

WHY does this technique exist?

Intuition The problem with code coverage

Coverage tells you a line was executed, not that it was checked. Consider:

def is_adult(age): return age >= 18

A test assert is_adult(20) == True runs the line — 100% line coverage! — but it would also pass if the code were age > 18, age >= 17, or even return True. Coverage gives you a false sense of safety.

WHY mutation testing fixes this: instead of asking "did we run the line?", it asks "if this line were wrong, would a test notice?" That is the real question a test suite should answer.

WHAT exactly is a mutant?

HOW the process runs (mechanically)

Worked example Walking the loop by hand

Original code:

def classify(n):
    if n > 0:          # line A
        return "pos"
    return "nonpos"

Test suite:

assert classify(5)  == "pos"
assert classify(-3) == "nonpos"

Mutant 1: change line A n > 0 → n >= 0. Why this mutant? The > vs >= boundary is a classic off-by-one bug. Run tests: classify(5)→"pos" ✓, classify(-3)→"nonpos" ✓. Both pass → mutant SURVIVED. 😱 Why did it survive? We never tested n = 0, the exact value where > and >= differ.

Fix: add assert classify(0) == "nonpos". Now the mutant returns "pos" for 0 → test fails → KILLED.

Mutant 2: change return "pos" → return "nonpos". Run tests: classify(5) now returns "nonpos" ≠ "pos" → test fails → KILLED immediately. Good, our suite already covers this.

Common mistakes (Steel-manned)

Forecast-then-Verify

Recall Before reading on: predict whether this mutant survives

Code: def max2(a,b): return a if a > b else b. Test: assert max2(3, 1) == 3. Mutant: a > b → a < b. Survive or killed?

Answer: With max2(3,1), mutant returns b = 1 ≠ 3 → test fails → KILLED. But mutant a > b → a >= b would survive this test (need a == b case, e.g. max2(5,5), to kill it).

Flashcards

What does mutation testing inject into the code?

Small deliberate bugs called mutants, each made by one mutation operator.

A mutant is "killed" when

at least one test in the suite fails on that mutant.

A mutant "survives" when

all tests still pass despite the injected bug — revealing a gap in the test suite.

Mutation score formula

K / (M - E)

= killed / (total mutants − equivalent mutants).

Why subtract equivalent mutants from the denominator?

They behave identically to the original, so no test can ever kill them; counting them would unfairly lower the score.

What's the key limitation of code coverage that mutation testing fixes?

Coverage proves a line ran, not that any assertion would catch a bug in it.

What is an equivalent mutant?

A syntactic change that produces identical behaviour to the original program; it can never be killed.

A surviving (non-equivalent) mutant tells you to do what?

Add or strengthen a test — not fix the production code.

Give a mutation operator example

Swap > → >=, + → -, True → False, or delete a statement.

Recall Feynman: explain to a 12-year-old

Imagine you built a smoke alarm and you want to know if it really works. You don't just look at it — you light a tiny match under it to see if it screams. Mutation testing does that to your tests: it secretly breaks your program in small ways (lights little "bug matches") and watches whether your tests scream (fail). If your tests stay silent while the program is broken, your tests are bad smoke alarms and need fixing.

Connections

Unit testing — mutation testing grades the unit tests you wrote.
Code coverage — the weaker metric mutation testing complements/exposes.
Test-driven development — TDD writes tests first; mutation testing audits their strength.
Boundary value analysis — many surviving mutants are boundary bugs ( $>$ vs $>=$ ).
Continuous integration — where mutation runs (often nightly, since it's slow).
Fault injection — the broader family of "deliberately break it and observe" techniques.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, mutation testing ka idea bahut simple hai. Tum code likhte ho, tests likhte ho, aur tests pass ho jaate hain — par sawaal yeh hai ki tumhare tests sach mein bugs pakadte hain ya bas dikhaawa hai? Code coverage sirf yeh batata hai ki line "chali" thi, par yeh nahi batata ki agar line galat hoti to koi test fail hota ya nahi. Yahin par mutation testing kaam aata hai.

Mutation testing kya karta hai? Woh tumhare code mein chhote-chhote jaan-boojh kar bugs daalta hai — jaise > ko >= bana dena, ya + ko -. Har aisa changed version ek mutant kehlata hai. Phir woh tumhare tests chalata hai. Agar koi test fail ho gaya, matlab tumne bug pakad liya — mutant killed (badhiya!). Agar saare tests phir bhi pass ho gaye, matlab tumhara test suite us bug ko miss kar gaya — mutant survived (yeh ek warning hai ki test kamzor hai).

Score nikaalne ka formula: $K / (M - E)$ , yaani killed mutants divide by (total mutants minus equivalent mutants). Equivalent mutants woh hote hain jo dikhne mein alag hain par behaviour bilkul same hai — unhe koi test maar hi nahi sakta, isliye unhe denominator se hata dete hain. 100% score milna mushkil hota hai, isliye 80/20 rule lagao: critical logic par focus karo.

Yaad rakho — agar mutant survive karta hai, to iska matlab tumhara production code kharab hai aisa nahi; iska matlab tumhe ek naya ya behtar test likhna hai. Mutation testing tumhare tests ko grade karta hai, code ko nahi. Isiliye yeh real software engineering mein quality ka sabse strong proof hai.

WHY does this technique exist?

WHAT exactly is a mutant?

HOW the process runs (mechanically)

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Software Engineering

Connections