Testing — unit, integration, system, acceptance, smoke, regression: Exercises

Two numbers we lean on again and again (from the parent note), restated so nothing is assumed:

Figure — Testing — unit, integration, system, acceptance, smoke, regression

Level 1 — Recognition

Goal: can you name the right test type and V&V question on sight?

L1.1

For each scenario, name the one test type (unit / integration / system / acceptance / smoke / regression):

You call parse_date("2024-13-01") alone and assert it raises an error — no DB, no network.
After merging a branch, CI re-runs the entire 300-test suite to make sure nothing that worked yesterday broke.
A stakeholder checks the delivered invoice screen against the user story "the customer wanted a PDF download button."
The pipeline's first 30-second job just verifies the app boots and the login page renders before anything else runs.
Your code writes a record through the real test database and reads it back to confirm the SQL and schema line up.
An automated browser clicks through checkout end-to-end on the fully assembled app.

Recall Solution L1.1

Unit — smallest isolated piece, no collaborators.
Regression — re-running existing tests after a change to protect old behaviour.
Acceptance — checks the right product was built (customer requirement / user story).
Smoke — tiny fast gate: "does it even turn on?"
Integration — real seam (code + real DB) tested together.
System — fully assembled app as a black box against requirements.

L1.2

Fill the V&V blank:

System test answers "are we building the product ____?"
Acceptance test answers "are we building the ____ product?"

Recall Solution L1.2

System test → right (this is Verification — meets the spec). See Verification and Validation.
Acceptance test → right product (this is Validation — meets the real need). Mnemonic: Verification = Vs the spec; Validation = Voice of the user.

Level 2 — Application

Goal: actually compute or write the thing.

A bug is worth $c = \$ 50 $i f c a ug h t ina u ni tt es t . I t esc a p es t o p r o d u c t i o n, cr oss in g$ n = 4 $s t a g e b o u n d a r i es (u ni t \to in t e g r a t i o n \to sy s t e m \to a cce pt an ce \to p r o d u c t i o n), w i t hm u l t i pl i er$ k = 10$ per stage. What does it cost in production?

Recall Solution L2.1

Apply $Cost (n) = c \cdot k^{n}$ . $Cost (4) = 50 \cdot 1 0^{4} = 50 \cdot 10000 = $500, 000.$ What this means: every boundary the bug slips past multiplies the blast radius (more people, rework, lost trust — see Defect Cost of Delay). A $50 miss becomes half a million dollars.

L2.2 — Suite trust

You have $n = 200$ independent tests, each returning the correct verdict with probability $p = 0.995$ . What is $P_{trust}$ , the chance the whole suite's "all green" can be believed?

Recall Solution L2.2

$P_{trust} = p^{n} = 0.99 5^{200} .$ Compute: $ln (0.995) \approx - 0.0050125$ , times $200 \approx - 1.0025$ , so $P_{trust} \approx e^{- 1.0025} \approx 0.367$ . $P_{trust} \approx 0.367 (\approx 37%) .$ Read it: roughly a 63% chance of a false red on any run. Flakiness compounds mercilessly — this is the arithmetic reason the pyramid keeps the count of flaky high-level tests tiny.

L2.3 — Write the asserts

Given def clamp(x, lo, hi): return max(lo, min(x, hi)), write a unit test covering: a value inside the range, a value below lo, a value above hi, and the boundary value equal to lo.

Recall Solution L2.3

def test_clamp():
    assert clamp(5, 0, 10) == 5    # inside → unchanged (happy path)
    assert clamp(-3, 0, 10) == 0   # below lo → clamped up
    assert clamp(42, 0, 10) == 10  # above hi → clamped down
    assert clamp(0, 0, 10) == 0    # boundary == lo → stays

Why four asserts? One case proves nothing. You cover the inside path, both clamp directions, and a boundary — the classic off-by-one danger zone. No DB, no network → genuinely a unit test.

Level 3 — Analysis

Goal: reason about why, compare, and diagnose.

L3.1 — Diagnose the failure

Your test_validate_email (a unit test) passes, but users still can't save. The repository.save(user) path silently corrupts data. Which test level would have caught this, and why did the unit test miss it?

Recall Solution L3.1

An integration test would have caught it. Why unit missed it: the unit test mocked the database away (see Mocks Stubs and Fakes), so it only proved the validation logic is correct — it never touched the real SQL, schema, or serialization. The corruption lives at the seam between code and DB, which a mock cannot exercise. Integration deliberately uses a real test DB precisely to test that seam.

L3.2 — Smoke vs regression

Both re-run a set of checks. Give the purpose and scope of each, and one sentence on why swapping them wastes resources.

Recall Solution L3.2

	Smoke	Regression
Purpose	Gate — is the build even alive?	Protect — did a change break old behaviour?
Scope	Breadth-first, shallow (a handful of critical paths)	The whole existing suite
When	Run first, fail fast	Run after a change, often every commit via Continuous Integration

Why swapping wastes resources: if you run the full 2-hour regression suite first and login was broken, you burned 2 hours to learn what a 30-second smoke test would have told you immediately. Smoke gates; regression guards.

L3.3 — Where does the flakiness live?

Using $P_{trust} = p^{n}$ , explain why moving 50 flaky end-to-end tests down into 50 fast, deterministic unit tests improves trust — even though $n$ is unchanged.

Recall Solution L3.3

$n$ (test count) is the same, but $p$ (per-test reliability) changes. Deterministic unit tests have $p$ very close to $1$ ; flaky end-to-end tests (timing, network, UI races) have $p$ noticeably below $1$ . Example: 50 flaky tests at $p = 0.98$ give $P = 0.9 8^{50} \approx 0.364$ . The same 50 as deterministic units at $p = 0.9999$ give $P = 0.999 9^{50} \approx 0.995$ . Lesson: the pyramid isn't only about speed — pushing work down raises $p$ , and since trust is $p^{n}$ , even a small rise in $p$ across many tests is a huge rise in $P_{trust}$ .

Level 4 — Synthesis

Goal: design a whole thing from the pieces.

L4.1 — Order a CI pipeline

You have five jobs with these run-times and failure likelihoods:

Job	Time	Historically fails
System (browser)	20 min	rarely
Unit	2 min	often (catches most bugs)
Smoke	30 s	occasionally (build broken)
Integration	5 min	sometimes
Acceptance (UAT)	manual	last

Order them for a fail-fast pipeline and justify with the cost-of-delay idea.

Recall Solution L4.1

$commit \to smoke (30s) \to unit (2m) \to integration (5m) \to system (20m) \to acceptance (UAT)$ Justification: order by cheapest-and-most-likely-to-fail first. Smoke gates for pennies of time; if the build is dead, stop before spending 27 minutes. Unit catches the most bugs cheapest, so it runs before the slow suites. System and manual UAT are last because they're expensive and least likely to be the first thing wrong. This is Defect Cost of Delay applied to compute time: every second a doomed pipeline keeps running is wasted cost.

L4.2 — Design a test plan for one feature

Feature: "user uploads a profile photo; it's resized and stored." Sketch one test of each level (unit, integration, system, acceptance) and say what each uniquely proves.

Recall Solution L4.2

Unit: resize(img, 200, 200) returns a 200×200 image (dependencies mocked). Proves: the resize math alone is correct.
Integration: store.save(resized_bytes) writes to the real test blob store and reads identical bytes back. Proves: the storage seam — encoding, keys, round-trip — works. See Mocks Stubs and Fakes.
System: an automated browser uploads a file end-to-end and the avatar appears on the profile page. Proves: the assembled app behaves per spec (Verification).
Acceptance: a stakeholder confirms the user story "as a user I can set a profile photo and see it everywhere" — the right feature (Validation), often written Given–When–Then (see Behaviour-Driven Development (BDD)). Each level proves something the level below cannot: logic → seam → whole system → right product.

Level 5 — Mastery

Goal: defend a trade-off with the numbers.

L5.1 — Justify the pyramid with both formulas at once

Team A writes 400 unit tests ( $p = 0.9999$ ) and 10 system tests ( $p = 0.97$ ). Team B writes 50 unit tests and 200 system tests (same $p$ 's). Compute each team's $P_{trust}$ and argue which suite you'd trust.

Recall Solution L5.1

Independence lets us multiply all per-test reliabilities: $P_{trust} = \prod p_{i} = p_{u}^{n_{u}} \cdot p_{s}^{n_{s}}$ .

Team A: $0.999 9^{400} \cdot 0.9 7^{10}$ . $0.999 9^{400} \approx e^{- 0.04} \approx 0.9608$ ; $0.9 7^{10} \approx 0.7374$ . Product $\approx 0.708$ .

Team B: $0.999 9^{50} \cdot 0.9 7^{200}$ . $0.999 9^{50} \approx 0.9950$ ; $0.9 7^{200} \approx e^{200 l n 0.97} = e^{- 6.093} \approx 0.00225$ . Product $\approx 0.00224$ .

Verdict: Team A's suite is trustworthy $\approx 71%$ of runs; Team B's a catastrophic $\approx 0.2%$ . The low- $p$ system tests, once you have many of them, torpedo $P_{trust}$ through the exponent $n_{s}$ . This is the quantitative case for the pyramid: keep $n$ small exactly where $p$ is low.

L5.2 — When is a low pyramid wrong?

Give a realistic case where you'd deliberately invest more in high-level tests than the pyramid suggests, and justify it against cost-of-delay.

Recall Solution L5.2

A payment/checkout flow or a safety-critical medical path. Here a bug that escapes to production has an enormous effective $c$ (real money lost, harm, legal cost) — so $Cost (n) = c \cdot k^{n}$ is dominated by a huge $c$ , not just $k^{n}$ . Spending on a few stable, well-maintained end-to-end tests over the critical journey is justified because the downside of missing it dwarfs their slowness and maintenance cost. The pyramid is a default, not a law: you shift effort toward wherever the product of probability-of-escape and cost-of-escape is highest. Code Coverage and Test-Driven Development (TDD) help, but coverage of the right high-value path beats raw coverage numbers.

Recall Self-test checklist

Name the six test types ::: unit, integration, system, acceptance, smoke, regression. System test → which V&V word? ::: Verification (build it right / meets spec). Acceptance test → which V&V word? ::: Validation (build the right thing / meets need). Cost of a bug escaping $n$ boundaries? ::: $Cost = c \cdot k^{n}$ ; $n$ counts boundaries, not stages. Trust of $n$ independent tests each reliable $p$ ? ::: $P_{trust} = p^{n}$ . Why not mock the real DB in a DB integration test? ::: You'd test the mock, not the seam — it becomes a unit test. Fail-fast pipeline order? ::: smoke → unit → integration → system → acceptance.

Exercises — Testing — unit, integration, system, acceptance, smoke, regression