Property-based testing

WHAT it is

Key vocabulary:

Property ::= a boolean rule true for every valid input.
Generator ::= code that produces random valid inputs (integers(), lists(), text()).
Shrinking ::= after finding a failing input, the framework simplifies it to the minimal example that still fails.

WHY: example tests vs property tests

Worked example Reversing a list

Example-based:

assert reverse([1,2,3]) == [3,2,1]   # one input, one output

Why weak? Passes even if reverse only works for length-3 lists.

Property-based (using Hypothesis):

from hypothesis import given
from hypothesis.strategies import lists, integers
 
@given(lists(integers()))
def test_reverse_twice_is_identity(xs):
    assert reverse(reverse(xs)) == xs    # property for ALL xs

Why this step? reverse(reverse(xs)) == xs is a mathematical truth independent of the values, so it must hold for every generated list — empty, single, huge, negative.

HOW to find good properties

You don't always know the exact output, but you usually know a relationship. Common patterns:

Formula Property patterns (derived from algebra, not memorized)

1. Round-trip (inverse functions). If $g$ undoes $f$ : $\forall x,\quad g(f(x)) = x$ Why: encode/decode, serialize/deserialize, compress/decompress are inverses by definition.

2. Idempotence. Applying twice = applying once: $f(f(x)) = f(x)$ Why: sort, normalize, dedupe — once done, redoing changes nothing.

3. Invariant / conservation. Some quantity is preserved: $\text{len}(\text{sort}(xs)) = \text{len}(xs)$ Why: sorting reorders but never adds/removes elements.

4. Oracle (test against a trusted slow version). $f_{\text{fast}}(x) = f_{\text{slow}}(x)$ Why: if a simple correct implementation exists, the property is "they agree."

5. Metamorphic (relate two related inputs). $\text{sort}(xs)=\text{sort}(\text{shuffle}(xs))$ Why: output shouldn't depend on input order.

HOW shrinking works (the killer feature)

Mental model of the shrink loop:

Found failing input $x_0$ .
Propose a "simpler" candidate $x'$ (drop an element, halve a number).
If $x'$ still fails → keep it, recurse. If it passes → discard, try another simplification.
Stop when no simpler candidate fails. Report that minimum.

This is a greedy descent toward the simplest failing case.

Worked example: a buggy `average`

def average(xs):
    return sum(xs) / len(xs)
 
@given(lists(integers()))
def test_average_within_bounds(xs):
    avg = average(xs)
    assert min(xs) <= avg <= max(xs)

Common mistakes (Steel-manned)

The 80/20

Recall Feynman: explain to a 12-year-old

Imagine you built a toy machine that flips a sock inside-out. Instead of testing it with three socks you picked, your robot friend grabs every random sock in the house — tiny socks, giant socks, holey socks — and flips each one. You don't tell the robot what the result should look like; you tell it ONE rule: "if you flip a sock and flip it again, it must look exactly like it started." The robot tries hundreds of socks. When one breaks the rule, it doesn't hand you the messy giant sock — it keeps swapping for simpler socks until it finds the tiniest one that still breaks the rule, then shows you THAT. Now you know exactly what's wrong.

Flashcards

What does a property-based test assert, unlike an example-based test?

A rule/invariant true for ALL valid inputs, not a single hardcoded input→output pair.

Define "shrinking" in property-based testing.

After finding a failing input, automatically simplifying it to the smallest/simplest input that still fails, for easy debugging.

Name the RIIOM property patterns.

Round-trip, Idempotence, Invariant, Oracle, Metamorphic.

Give the round-trip property formula.

g(f(x)) == x for all x, where g is the inverse of f (e.g. decode(encode(x))).

Write the idempotence property.

f(f(x)) == f(x) (e.g. sort, normalize, dedupe).

Why not assert an exact expected output inside @given?

You'd have to reimplement the function to know the answer for random inputs, copying its bugs; assert a relationship instead.

What is a "generator/strategy"?

Code that produces random VALID inputs of a given type (integers, lists, text).

Is passing 1000 random cases a correctness proof?

No — it's sampling that raises confidence and finds bugs, not exhaustive proof.

An invariant property for sort?

len(sort(xs)) == len(xs) and sorted output is a permutation of input.

What's the "oracle" pattern?

Compare the fast implementation against a simple trusted (slow) reference: f_fast(x) == f_slow(x).

Why must PBT runs be deterministic per input?

So failures reproduce; hidden randomness/global state causes flaky, non-reproducible failures.

Connections

Unit testing — PBT complements, doesn't replace, example tests.
Test-driven development — properties can drive design.
Invariants and contracts — properties ARE runtime-checkable invariants.
Fuzzing — random input generation cousin (PBT = fuzzing + structured assertions + shrinking).
Hypothesis (Python) / QuickCheck (Haskell) — the original PBT tools.
Pure functions — easiest things to property-test (deterministic, no side effects).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, normal testing mein hum khud 2-3 examples sochte hain: "agar input [1,2,3] hai to output [3,2,1] aana chahiye." Problem ye hai ki hum sirf wahi inputs test karte hain jo humein yaad aate hain. Jo edge cases hum bhool jaate hain (empty list, negative numbers, bahut bade numbers) — bug wahin chhupa hota hai. Property-based testing iska solution hai: tum ek rule (property) likhte ho jo har valid input ke liye sach hona chahiye, aur framework (jaise Python ka Hypothesis) sainkdon random inputs bana ke us rule ko todne ki koshish karta hai.

Rule kaise socho? Output ka exact value pata nahi hota, par relationship pata hota hai. Jaise: reverse(reverse(xs)) == xs — list ko do baar palto to wahi wapas aa jaayegi, chahe values kuch bhi ho. Ye RIIOM patterns yaad rakho — Round-trip (encode/decode wapas same), Idempotence (sort do baar = ek baar), Invariant (sort karne ke baad length same), Oracle (slow but correct version se compare), Metamorphic (input ka order change karo, output same). In paanch patterns se 80% real cases cover ho jaate hain.

Sabse mast feature hai shrinking. Maan lo framework ne fail karne wala input dhoondha [847,-3,901,0,-12] — itna ganda input dekh ke debug karna mushkil. To framework khud ise chhota karta jaata hai, har baar simpler input try karke jo abhi bhi fail kare, aur ant mein tumhe minimal example deta hai jaise [0,-1]. Isse bug seedha samajh aa jaata hai.

Ek important baat: PBT 100 cases pass kar gaya iska matlab code "proven correct" nahi hai — ye sampling hai, proof nahi. Lekin ye un bugs ko pakadta hai jo tum kabhi haath se na sochte. Isliye example tests + property tests dono saath use karo. Aur dhyan rakho generator sirf valid inputs banaye, warna false failures aayenge.