Property-based testing: Question bank — Coding deep dive

True or false — justify

True or false: A property-based test replaces the need for example-based tests.

False — it complements them. You still want Unit testing examples for known-tricky cases (off-by-one, that one Unicode string) because a random sampler may never hit exactly them.

True or false: If a property test passes 10,000 random inputs, the function is proven correct.

False — PBT is sampling, not proof. The input space is usually infinite; passing raises confidence and finds bugs but never visits every input.

True or false: reverse(reverse(xs)) == xs is a stronger check than reverse([1,2,3]) == [3,2,1].

True — the round-trip property must hold for every list, so it can't be satisfied by code that only special-cases length-3 lists, whereas the single example can.

True or false: A generator should produce as many wild, unconstrained inputs as possible for maximum coverage.

False — a generator must produce only valid inputs. If your function requires sorted lists, feeding it unsorted garbage produces false failures that test nothing real.

True or false: Shrinking changes which bug is present.

False — shrinking only simplifies the input that triggers an already-found failure. The bug is the same; you just get a smaller, more readable counterexample like [0, -1] instead of [847, -3, 901, ...].

True or false: An idempotence property f(f(x)) == f(x) is a good fit for a random-number generator.

False — a good idempotence candidate is sort, normalize, or dedupe, where redoing changes nothing. A random generator is supposed to give different output, so it violates idempotence by design.

True or false: The oracle pattern requires you to already know the exact expected output for each input.

False — it only requires a trusted (usually slow/simple) reference implementation; the property is "fast and slow agree," so the reference computes the expected value for you.

True or false: Metamorphic properties relate the output to a hardcoded value.

False — they relate the outputs of two related inputs (e.g. sort(xs) == sort(shuffle(xs))), with no hardcoded expected value at all.

True or false: If a property test is flaky, the fix is usually to add more retries.

False — flakiness comes from hidden randomness or global state making the same input behave differently. The fix is determinism per input (seed it, isolate state), not masking it with retries.

Spot the error

Spot the error: @given(lists(integers())) \n def t(xs): assert average(xs) == sum(xs)/len(xs).

You've reimplemented the function inside the test, so it can't catch a bug in average — the test would copy the same mistake. Assert a relationship (min(xs) <= avg <= max(xs)), not the implementation.

Spot the error: A test uses @given(lists(integers())) on a function that only accepts non-empty lists, then complains about a ZeroDivisionError.

The failure is a generator problem, not a code bug for that contract — either constrain the generator with min_size=1 or decide empty input is valid and make the function raise a clear error. The framework did its job by surfacing the ambiguity.

Spot the error: Someone writes @given(sorted_lists()) \n def t(xs): assert binary_search(xs, xs[0]) == 0 but the generator can emit [].

Accessing xs[0] on an empty list crashes the test itself, masking whatever it meant to check. Either add assume(len(xs) > 0) or a min_size, so degenerate inputs are handled before the property.

Spot the error: A "property" reads assert result is not None.

That's barely a property — nearly any wrong output satisfies it. Strong properties capture a meaningful relationship (round-trip, invariant, oracle); "not None" tests almost nothing.

Spot the error: To speed things up, a developer wraps the whole property body in try: ... except: pass.

Swallowing exceptions means real failures (including the very bugs PBT exists to find) silently pass. Let the framework see the exception — it will record and shrink the failing seed for you.

Spot the error: A property compares floating-point results with == after two different computation orders.

Floating-point isn't associative, so == will report false failures. Use a tolerance (abs(a-b) < eps) — this is the "make the property tolerant" fix for non-exact arithmetic.

Why questions

Why do random inputs find bugs that hand-picked examples miss?

You only write examples for inputs you imagine; bugs hide in inputs you forgot — empty list, negative zero, huge numbers, Unicode. A generator explores the space you didn't think of.

Why is shrinking called a "greedy descent"?

It repeatedly takes a simpler candidate (fewer elements, values nearer zero) and keeps it only if it still fails, marching monotonically toward the minimal failing case and stopping when no simpler candidate fails.

Why prefer asserting a relationship over asserting an exact output?

For a random input you don't know the exact answer without recomputing it, and recomputing means re-implementing (and re-bugging) the function. Relationships like g(f(x)) == x are true regardless of the value.

Why does the round-trip pattern fit encode/decode and serialize/deserialize?

Those operations are inverse functions by definition — decoding is meant to undo encoding — so decode(encode(x)) == x is a mathematical truth that must hold for every input.

Why must generators emit only valid inputs?

A property is a rule for valid inputs; feeding invalid data tests undefined behaviour and produces false alarms that waste debugging time and erode trust in the suite (see the "huge unconstrained generators" Fuzzing contrast — pure fuzzing skips the validity structure).

Why does Hypothesis replay a failing seed on the next run?

So failures are reproducible. Without replay a random failure might vanish on the next run, making it impossible to confirm a fix.

Why do Pure functions make especially good PBT targets?

They depend only on their inputs and have no side effects, so the same input always gives the same output — exactly the determinism PBT needs for reproducible, shrinkable failures.

Edge cases

Edge case: What input breaks average(xs) = sum(xs)/len(xs) and why is it the minimal failing case?

The empty list [] — len==0 gives a ZeroDivisionError. It's already minimal, so shrinking can't simplify further and reports xs=[] directly.

Edge case: For the invariant min(xs) <= avg <= max(xs), name a degenerate input that still satisfies it and one that breaks the property's assumptions.

A single-element list [k] satisfies it (min==avg==max==k); the empty list breaks the assumptions because min, max, and the division are all undefined — signalling you must decide the empty-list contract.

Edge case: Why is a one-element list a sneaky trap for reverse(reverse(xs)) == xs?

It's not — it passes trivially, which is exactly the danger: a function that only handles length-≤1 lists would pass this single case, so you rely on the generator also producing longer and empty lists to exercise the real logic.

Edge case: For sort, what boundary inputs should you trust the generator to include, and which invariants stay true on them?

The empty list, a single element, an already-sorted list, a reverse-sorted list, and lists with duplicates. On all of them len(sort(xs)) == len(xs) and the output remains a permutation of the input — the invariant never degenerates.

Edge case: A property involves random.shuffle inside the code under test. What edge behaviour makes the test flaky and how is it tamed?

For the same input the code produces different orders, so an equality property fails unpredictably. Seed the shuffle (or assert an order-independent invariant like "same multiset"), restoring determinism per input.

Recall One-line summary of the whole trap set

Properties assert relationships for all valid inputs; generators must stay valid; shrinking minimises the input, not the bug; passing is confidence, not proof; and every function needs its degenerate inputs (empty, single, zero) explicitly reasoned about.

Connections

Property-based testing — the parent topic these traps drill.
Unit testing — PBT complements example tests, never replaces them.
Invariants and contracts — properties ARE runtime-checkable invariants.
Fuzzing — the validity-constraint traps sharpen the PBT-vs-fuzzing line.
Pure functions — why determinism makes a good PBT target.