This page is a drill . The parent note told you what the six test types are . Here we throw every awkward case at you — every "which test do I use?", every degenerate input, every limiting number — and work each one fully. First guess, then read.
Intuition How to read this page
The parent gave you two formulas and six definitions. A definition you can recite is not a definition you can use . So below, every "cell" of the scenario matrix gets its own worked example. When a cell involves numbers (cost, probability, coverage), the exact number is machine-checked in the appendix — you can trust it.
Think of every question this topic can ask as living in one of these cells. The worked examples that follow each carry a [Cell X] tag so you can see the whole space get covered.
Cell
Case class
What makes it tricky
Example
A
Classify a test (unit vs integration)
the "mocked DB" trap
Ex 1
B
Classify a test (system vs acceptance)
Verification vs Validation
Ex 2
C
Classify a test (smoke vs regression)
both "re-run checks"
Ex 3
D
Cost-of-delay — normal n
Cost = c k n
Ex 4
E
Cost-of-delay — degenerate n = 0
"caught immediately" edge
Ex 4
F
Cost-of-delay — limiting large n
why cost explodes
Ex 4
G
Suite trust — normal case
P = p n
Ex 5
H
Suite trust — degenerate p = 1 or n = 1
perfect / single test
Ex 5
I
Suite trust — limiting n → ∞
why flaky top of pyramid is poison
Ex 5
J
Real-world word problem
pick tests + estimate savings
Ex 6
K
Exam-style twist
"all tests green" but wrong product
Ex 7
L
Ordering a pipeline
fail-fast arithmetic
Ex 8
The two formulas you will lean on (from the parent, restated so nothing is assumed):
Worked example Example 1 —
[Cell A] unit vs integration: the mocked-DB trap
Statement. A teammate writes: "I test repository.save(user) — but I replace the database with a mock that just records the call." They call it an integration test . Are they right?
Forecast: unit or integration? Say your answer before reading.
Ask what the test actually exercises. Why this step? A test's type is decided by what runs for real , not by the function's name. Here the real database is swapped for a mock — so the only real code running is save's own logic.
Check the seam. Why this step? Integration's whole job is the seam — the SQL string, the schema, the serialization between your object and the table. A mock returns a canned "ok" and touches none of that.
Conclude. With the DB mocked, no seam is tested → this is a unit test wearing an integration costume.
Verify: flip it — remove the mock, point at a real test database, write the row, read it back. Now the schema and SQL run for real. That is the integration test. The presence of a real collaborator at the boundary is the deciding evidence. ✔
Worked example Example 2 —
[Cell B] system vs acceptance: same app, different question
Statement. The app is fully assembled. Test 1: "Enter a valid password → user is logged in and redirected to /home, matching spec section 4.2." Test 2: "The client said login should also let staff badge-in with an ID card; does it?" Which is system , which is acceptance ?
Forecast: which is which?
Read Test 1's yardstick. Why this step? It checks against the written spec ("section 4.2"). That is Verification — "did we build it right ?" → system test .
Read Test 2's yardstick. Why this step? It checks against the client's real need (badge-in), regardless of whether the spec mentioned it. That is Validation — "did we build the right thing?" → acceptance test .
Notice both are end-to-end and black-box. Why this step? That shared surface is exactly why students confuse them — the difference is the reference standard , not the mechanics.
Verify: a system can pass every Test-1-style check and still fail Test 2 — spec-perfect but wrong product. That impossibility-to-collapse-them proves they are distinct. ✔
Worked example Example 3 —
[Cell C] smoke vs regression: both "re-run checks"
Statement. After a code change, a CI server (see Continuous Integration ) runs: (a) a 30-second set — "app boots, homepage 200, login works" — before anything else; (b) the full 4000-test suite that has accumulated over two years. Label each.
Forecast: which is smoke, which is regression?
Look at set (a)'s timing and breadth. Why this step? Run first , shallow, few critical checks, purpose = gate ("is it even standing?"). That is a smoke test .
Look at set (b)'s purpose. Why this step? Re-running old tests to be sure the change didn't break previously-working features → regression suite .
Separate purpose from mechanism. Why this step? Both "re-run checks", yes — but smoke is breadth-first gate , regression is protect existing behaviour . Different intent.
Verify: if smoke (a) fails, you stop and never run (b). If it were "just a small regression", stopping would make no sense. The gating behaviour confirms (a) is smoke. ✔
The figure below draws Cost ( n ) = c k n with c = 1 , k = 10 . Notice the vertical axis is logarithmic — a straight line there means explosive growth.
Worked example Example 4 —
[Cells D, E, F] cost of a bug at n = 0 , n = 3 , and large n
Statement. A bug is born in the requirements stage costing c=\ 50. C os t m u l t i pl i es b y k=10$ per stage it escapes. Find the cost if caught (E ) immediately, (D ) after escaping 3 stages, and (F ) reason about escaping 6 stages.
Forecast: guess the three numbers — most people badly underestimate the last.
Case n = 0 (Cell E, degenerate "caught immediately"). Why this step? You must handle the edge where the bug escapes zero stages. k 0 = 1 for any k (multiplying k by itself zero times leaves you at 1). So \text{Cost}(0)=50\cdot 10^0 = 50\cdot1 = \ 50$. The formula correctly says "no escape, no penalty."
Case n = 3 (Cell D, normal). Why this step? This is the everyday case in the parent note. \text{Cost}(3)=50\cdot10^3 = 50\cdot 1000 = \ 50{,}000$.
Case n = 6 (Cell F, limiting). Why this step? To feel why "shift left" matters, push n up. \text{Cost}(6)=50\cdot10^6 = \ 50{,}000{,}000$. Look at the figure: on the log axis this is a straight climb, meaning each stage is a fixed multiplicative jump, not additive.
Verify: ratio check — going from n = 0 to n = 3 should multiply by k 3 = 1000 : 50 , 000/50 = 1000 . ✔ And n = 3 to n = 6 multiplies by another 1000 : 50 , 000 , 000/50 , 000 = 1000 . ✔ Units stay dollars throughout because k is unitless. See Defect Cost of Delay for the empirical origin of k ≈ 10 .
The next figure plots P trust = p n against the number of tests n , for three reliabilities p . Watch the top curve collapse as tests accumulate.
Worked example Example 5 —
[Cells G, H, I] trust for normal, perfect, and huge suites
Statement. Each test independently gives the right answer with probability p . Compute the chance of a trustworthy "all green": (H , degenerate) a single perfect test n = 1 , p = 1 ; (H , degenerate) a single flaky test n = 1 , p = 0.999 ; (G , normal) n = 500 , p = 0.999 ; (I , limiting) reason about n → very large.
Forecast: for n = 500 each 99.9% reliable — is the suite still ~99%? Guess before computing.
n = 1 , p = 1 (Cell H, perfect). Why this step? Establish the ceiling. P = 1 1 = 1 — a perfect single test never lies. Sanity anchor.
n = 1 , p = 0.999 (Cell H, single flaky). Why this step? With one test the suite trust equals the test's reliability: P = 0.99 9 1 = 0.999 . So one flaky test is barely felt.
n = 500 , p = 0.999 (Cell G, the shock). Why this step? Independence means we multiply 0.999 by itself 500 times: P = 0.99 9 500 ≈ 0.607 . Only ~61% chance the whole board is honestly green — a ~39% chance of a false red . This is the number that surprises everyone.
n → ∞ (Cell I, limiting). Why this step? Any p < 1 raised to ever-larger n drives p n → 0 . So an unbounded pile of even slightly-flaky tests trends toward "never trustworthy." That is precisely why the pyramid keeps the count of flaky high-level tests small .
Verify: monotonic sanity — adding tests should never raise trust when p < 1 . Indeed 0.99 9 500 < 0.99 9 1 < 1 1 . ✔ And with truly perfect tests (p = 1 ) the size n is irrelevant: 1 500 = 1 . ✔ See Code Coverage — coverage counts how much code tests touch, a separate axis from how reliably they report.
Worked example Example 6 —
[Cell J] "should we buy the CI runner?"
Statement. Your team ships ~20 bugs/year. Historically each escapes to production (n = 4 stages from where it could have been caught by an automated suite) costing on average c=\ 100a t in t r o d u c t i o n , k=10p er s t a g e . A n e w [[ C o n t in u o u s I n t e g r a t i o n ]] r e g r ess i o n se t u pw o u l d c a t c h ∗ ∗ 75% ∗ ∗ o f t h e ma t s t a g e 0. I t cos t s $40{,}000$/year. Worth it?
Forecast: save money or waste it? Guess the order of magnitude first.
Cost per escaped bug today. Why this step? Baseline. \text{Cost}(4)=100\cdot10^4 = \ 1{,}000{,}000$ per bug that reaches production.
Bugs the CI catches. Why this step? 75% of 20 = 15 bugs now caught at stage 0, each costing only \text{Cost}(0)=100\cdot10^0=\ 100$.
Money saved on those 15. Why this step? Each saved bug's saving is 1{,}000{,}000 - 100 = \ 999{,}900. T o t a l : 15 \times 999{,}900 = $14{,}998{,}500$.
Net after the tool's price. Why this step? Subtract the \ 40{,}000cos t : 14{,}998{,}500 - 40{,}000 = $14{,}958{,}500$ net saving. Overwhelmingly worth it.
Verify: dimension check — dollars minus dollars = dollars. ✔ Ballpark: 15 bugs \times \sim\ 1\text{M} \approx $15\text{M}s a v e d a g ain s t a $40\text{k}cos t , a 375 × r e t u r n . T h e d o minan ceo f k^n$ makes the tool cost a rounding error. ✔
Worked example Example 7 —
[Cell K] "every test is green, ship it?"
Statement. Exam prompt: "A payroll system passes 100% of its unit, integration, AND system tests. The QA lead still refuses to sign off. Give a scenario where she is right, and name the test that would catch it."
Forecast: can a fully-green build still be wrong? Decide yes/no first.
Recall what each passed test proves. Why this step? Unit/integration/system all check the app against its own spec — that is Verification (built right ). None of them ask whether the spec matches reality.
Construct the gap. Why this step? Suppose the spec said "tax = flat 20%", but the client's legal requirement is a progressive bracket. Every test encodes the wrong 20% and passes — the product is spec-perfect yet wrong .
Name the catching test. Why this step? Only Validation — an acceptance test / UAT driven by the client's real requirement — exposes it, because it references the need , not the spec.
Verify: consistency with Example 2 — this is Cell B's principle in reverse. "Green system tests" ⇏ "right product", exactly because system = Verification, acceptance = Validation. The QA lead is right. ✔ (Related: Behaviour-Driven Development (BDD) writes acceptance criteria as Given–When–Then to close this gap early.)
Worked example Example 8 —
[Cell L] why smoke goes first
Statement. A pipeline has: smoke (30 s), unit (2 min), integration (5 min), system (20 min). Suppose a broken build fails at the login step. Compare wasted time for two orderings: (a) smoke → unit → integration → system, vs (b) system → integration → unit → smoke.
Forecast: how many minutes does the wrong order waste before it discovers the same failure?
Order (a): smoke first. Why this step? Smoke's job is to catch "login broken" in 30 s. Login fails at 30 s → we stop. Wasted work beyond the finding: 0 . Total spent: 0.5 min.
Order (b): system first. Why this step? Now the cheap login check is buried last. We grind through system (20) + integration (5) + unit (2) = 27 min before the tiny smoke check finally flags login. Total: 27 + 0.5 = 27.5 min.
Compare. Why this step? The two orders find the same defect; order (a) does it in 0.5 min, order (b) in 27.5 min — a wasted 27 min per failed build.
Verify: arithmetic — 20 + 5 + 2 = 27 min of avoidable work; 27.5 − 0.5 = 27 . ✔ This is the parent note's "cheapest-and-most-likely-to-fail first" principle turned into a stopwatch. ✔
Recall Quick self-test on the matrix
Test with mocked DB is which type? ::: A unit test — no real seam runs.
System test answers which V&V word? ::: Verification — built the product right.
Acceptance test answers which V&V word? ::: Validation — built the right product.
Cost of a bug caught immediately, c=\ 50, k=10? ::: 50\cdot10^0 = $50. C os t a f t er esc a p in g 3 s t a g es , s am e n u mb er s ? ::: 50\cdot10^3 = $50{,}000. S u i t e t r u s t f or n=500t es t se a c h p=0.999? ::: 0.999^{500}\approx 0.61— ab o u t 39% f a l se − r e d r i s k . A s n\to\inftyw i t han y p<1, P_{\text{trust}}\to? ::: 0$ — flaky tests compound to worthlessness.
Why does smoke run first in a pipeline? ::: To fail-fast and avoid wasting minutes on the full suite.