Worked examples — Floating point gotchas — catastrophic cancellation, associativity failure

Before anything: three words we will reuse.

---

The scenario matrix

Every trouble on this topic falls into one of these cells. Each row is one worked example below.

#	Case class	Trigger	What breaks	Fix family
A	Subtract near-equal positives	$a\approx b$, both $>0$	catastrophic cancellation	algebraic reformulation
B	Quadratic, same-sign root	$b^2\gg 4ac$, take safe sign	fine (no fix needed)	—
C	Quadratic, opposite-sign root	the other root	cancellation in $-b+\sqrt{\cdot }$	$x=c/(a{x}_{\text{safe}})$
D	Degenerate: term rounds to zero	$1-\cos x$, $x\to 0$	total loss (answer = 0)	trig identity
E	Absorption — small + huge	$1+10^{16}$	the small one vanishes	reorder / Kahan
F	Associativity flip	$(a+b)+c\ne a+(b+c)$	grouping changes answer	cancel large terms first
G	Summation order over a list	many tiny + one huge	error grows with $n$	sort ascending / Kahan
H	Equality comparison	0.1+0.2 == 0.3	False unexpectedly	tolerance compare
I	Real-world word problem	GPS / distance difference	cancellation hides in disguise	reformulate the physics
J	Exam twist	"higher precision fixes it?"	conditioning ≠ precision	reformulate, not upgrade

---

Example A — subtract near-equal positives

Steps.

1. True value. $\sqrt{x+1}-\sqrt{x}\approx \frac{1}{2\sqrt{x}}=\frac{1}{2\times 10^8}=5\times 10^{-9}$. Why this step? We need the honest target so we can measure error against it. The approximation comes from $\sqrt{x+1}-\sqrt{x}=\frac{1}{\sqrt{x+1}+\sqrt{x}}\approx \frac{1}{2\sqrt{x}}$.

2. Naive compute. In doubles, $\sqrt{10^{16}}=10^8$ exactly, and $\sqrt{10^{16}+1}$ — but $10^{16}+1$ rounds to $10^{16}$ (the gap near $10^{16}$ is $2$, so adding $1$ falls off the edge, cell E in disguise). So both square roots equal $10^8$, and the subtraction gives $\hat{f}=0$. Why this step? Shows the double failure: absorption inside, then subtracting two identical numbers gives exactly $0$.

3. Measure the damage. Relative error $=\left|\frac{0-5\times 10^{-9}}{5\times 10^{-9}}\right|=1$ — 100% wrong. Why this step? Confirms "catastrophic": the answer is entirely lost.

4. Reformulate. Multiply top and bottom by the conjugate: $\sqrt{x+1}-\sqrt{x}=\frac{(\sqrt{x+1}-\sqrt{x})(\sqrt{x+1}+\sqrt{x})}{\sqrt{x+1}+\sqrt{x}}=\frac{1}{\sqrt{x+1}+\sqrt{x}}.$ Now compute $\frac{1}{\sqrt{10^{16}+1}+\sqrt{10^{16}}}=\frac{1}{2\times 10^8}=5\times 10^{-9}$. No subtraction of near-equals — we only add same-sign roots and divide. Why this step? Division and same-sign addition never amplify relative error (parent note, §2), so we route around the villain.

Verify: the reformulated value $5\times 10^{-9}$ matches the true $\frac{1}{2\sqrt{x}}$ to full precision. Sanity: for large $x$ the gap between consecutive roots should shrink like $1/\sqrt{x}$ — it does. ✓

---

Examples B & C — the two quadratic roots

We take one quadratic and extract both roots, because the two roots live in different matrix cells: one is safe (B), one cancels (C).

Steps.

1. Discriminant. $b^2-4ac=4\times 10^{10}-4=39999999996$, so $\sqrt{\cdot }\approx 199999.99999$. Why this step? Both roots share this square root; compute once.

2. Same-sign root (Cell B — safe). Since $b<0$, the term $-b=+200000$ is positive; the safe root adds two positives: $x_1=\frac{-b+\sqrt{b^2-4ac}}{2a}=\frac{200000+199999.99999}{2}\approx \mathrm{199999.999995.}$ Why this step? Adding two same-sign near-equal numbers does not cancel — no amplification. This root is trustworthy.

3. Opposite-sign root, naive (Cell C — catastrophic). $x_2^{\text{naive}}=\frac{200000-199999.99999}{2}=\frac{0.00001}{2}\approx 5\times 10^{-6}.$ Here the cancellation happens between two operands $p=200000$ and $q=199999.99999$. The parent note's amplifier, written for a generic subtraction $p-q$, is $\frac{|p|+|q|}{|p-q|}$ (the letters $a,b$ in §2 of the parent are just placeholder operands — not the quadratic coefficients here). Plugging in: $\frac{|p|+|q|}{|p-q|}\approx \frac{400000}{10^{-5}}=4\times 10^{10}$ — precision annihilated. Why this step? Exposes the cancellation and its size, being careful not to confuse the operands $p,q$ with the coefficients $a,b,c$.

4. Stable second root. Use $x_1{x}_2=c/a$ (product of roots), so $x_2=\frac{c}{a{x}_1}=\frac{1}{199999.999995}\approx 5.00000000013\times 10^{-6}.$ Why this step? We already have the safe root $x_1$; the product identity gives $x_2$ using only division — no subtraction of near-equals. See Quadratic formula numerical issues.

Verify: plug $x_2$ back: $x_2^2-200000x_2+1\approx 0$. Product check: $x_1{x}_2=c/a=1$. Sum check: $x_1+x_2=-b/a=200000$. ✓

---

Example D — a term rounds to zero (degenerate)

Steps.

1. True value. Taylor: $\cos x=1-\frac{x^2}{2}+\dots$, so $1-\cos x=\frac{x^2}{2}$ and $f\to \frac{1}{2}$. Why this step? Establishes the target $0.5$.

2. What the machine stores. $\cos (10^{-8})=1-5\times 10^{-17}+\dots$. But $5\times 10^{-17}<u\approx 1.11\times 10^{-16}$, so $\cos (10^{-8})$ rounds to exactly $1.0$. Why this step? The information that made $f$ nonzero lives below machine epsilon — it is gone before subtraction.

3. Naive result. $1-1=0$, so $\hat{f}=0/x^2=0$. Relative error $=1$ (total loss). Why this step? Confirms degeneracy: the answer collapses to zero.

4. Reformulate with a trig identity. $1-\cos x=2{\sin }^2(x/2)$, hence $f(x)=\frac{2{\sin }^2(x/2)}{x^2}=\frac{1}{2}{\left(\frac{\sin (x/2)}{x/2}\right)}^2.$ $\sin$ of a small argument is computed accurately (it's $\approx$ the argument, no cancellation), and $\frac{\sin t}{t}\to 1$, so $f\to \frac{1}{2}$. Why this step? We replaced "subtract near-equals" by "$\sin$ of a small number", which the library evaluates faithfully.

Verify: at $x=10^{-8}$, $\sin (5\times 10^{-9})/5\times 10^{-9}\approx 1$, so $f\approx \frac{1}{2}(1{)}^2=0.5$. Matches the limit. ✓

The figure below plots both the naive and reformulated versions across many $x$. Read it like this: the dashed gray line is the true limit $0.5$; the green curve (reformulated) hugs it all the way down; the red curve (naive) rides along $0.5$ until $x\approx 10^{-8}$, then falls off a cliff to $0$ the moment $\cos x$ rounds to $1$. That cliff is the catastrophic cancellation — the picture shows exactly where in $x$ the naive formula dies.

---

Example E — absorption (small + huge)

Steps.

1. Gap at $10^{16}$. Near a value of magnitude $2^{53}\approx 9\times 10^{15}$, consecutive doubles differ by $2$ (the mantissa's last bit now weighs $2$, not $1$). So $10^{16}$ has neighbours $2$ apart. Why this step? We must know the resolution at that magnitude — precision is relative, so big numbers have coarse spacing (see IEEE-754 floating point representation).

2. Add the 1. $10^{16}+1=10000000000000001$ (odd) — but that's not representable; the nearest doubles are $10^{16}$ and $10^{16}+2$. Rounding-to-nearest keeps $10^{16}$. Why this step? The $+1$ is smaller than half the gap, so it is absorbed — vanishes entirely.

3. Consequence. $\widehat{(1+10^{16})}=10^{16}$. The 1 left no trace. Why this step? This is the mechanism behind the associativity break in Example F.

Verify: in Python, 1e16 + 1 == 1e16 is True; 1e16 + 2 == 1e16 is False. The gap is exactly $2$. ✓

---

Example F — associativity flip

Before the example, one small lemma we will lean on.

Steps.

1. Left grouping $(a+b)+c$. $a+b=1+10^{16}\to 10^{16}$ (absorption, Example E). Then $+c=10^{16}-10^{16}=0$. Why this step? Shows the 1 dies in the first add, so the result is $0$.

2. Right grouping $a+(b+c)$. $b+c=10^{16}-10^{16}=0$ exactly — this is the Sterbenz lemma above: $b$ and $-c$ are equal in magnitude, so the subtraction introduces no rounding. Then $+a=0+1=1$. Why this step? Here the huge numbers cancel first (exactly), protecting the tiny $1$.

3. Compare. Left $=0$, right $=1$. Differ by 100%. Why this step? Demonstrates floating addition is not associative — grouping decides which bits survive.

4. Rule of thumb. Cancel large opposite-sign terms before adding small ones. Why this step? Prevents the small term being absorbed.

Verify: (1.0 + 1e16) + (-1e16) == 0.0 and 1.0 + (1e16 + (-1e16)) == 1.0. ✓

The bar chart below makes the flip vivid. Read it like this: the left red bar is the $(a+b)+c$ grouping — it collapses to $0$ because the $1$ was absorbed by $10^{16}$ before $c$ could cancel it; the right green bar is $a+(b+c)$ — it reaches $1$ because the two huge terms annihilate first (Sterbenz-exact) and the $1$ survives. Two identical sets of numbers, two different heights: that height gap is associativity failure.

---

Example G — summation order over a list

Steps.

1. Naive left-to-right. Start $s=10^{16}$. Add the first $1$: $10^{16}+1\to 10^{16}$ (absorbed). Every subsequent $+1$ is likewise absorbed. Final $\hat{s}=10^{16}$ — all $10^6$ ones lost. Why this step? The running partial sum is huge from step one, so each tiny addend falls below its gap.

2. Ascending order. Sort so the huge number is last: sum the million ones first, getting $10^6$ (each add is between comparable numbers — safe), then add $10^{16}$. Now $10^{16}+10^6$: the gap at $10^{16}$ is $2$, and $10^6\gg 2$, so it survives: $\hat{s}=10000000001000000$. Why this step? Adding smallest first lets them accumulate into a number big enough not to be absorbed. See Round-off error propagation.

3. Kahan option. Kahan compensated summation carries the lost low bits in a compensator c, giving error $\sim u$ independent of order and of $n$. Why this step? When you can't sort (streaming data), Kahan recovers accuracy anyway.

Verify: naive gives $10^{16}$; ascending gives $10^{16}+10^6$; both compared to the true $10000000001000000$. ✓

---

Example H — the equality trap

Steps.

1. Why they aren't exact. $0.1,0.2,0.3$ are not finite binary fractions (like $1/3$ in decimal). Each is stored as the nearest double, already off by $\sim u$. Why this step? The premise "equal in math ⇒ equal in floats" is false because the inputs themselves are rounded.

2. The sum. $\text{fl}(0.1)+\text{fl}(0.2)=0.30000000000000004\dots$, while $\text{fl}(0.3)=0.29999999999999998\dots$. They differ by $\sim 5.6\times 10^{-17}$. Why this step? Shows the two sides land on different doubles, so == returns False.

3. Fix: tolerance compare. Use math.isclose(x, y), i.e. $|x-y|\le \text{atol}+\text{rtol}\cdot \max (|x|,|y|)$ with default rtol=1e-9. Why this step? We ask "are they equal within round-off?" instead of "bit-identical?".

Verify: 0.1 + 0.2 == 0.3 is False; the difference is about $5.55\times 10^{-17}$; abs((0.1+0.2)-0.3) < 1e-9 is True. ✓

---

Example I — real-world word problem

Steps.

1. True difference. $\Delta =6378137.250-6378137.000=0.250\text{ m}$. Why this step? Sets the target: a 25 cm difference.

2. Digit accounting. Each altitude is good to $\sim 16$ significant digits (double). Absolute error of each is about $|R|\times u\approx 6.378\times 10^6\times 1.11\times 10^{-16}\approx 7\times 10^{-10}\text{ m}$. Subtraction is exact (Sterbenz — the two values are within a factor of 2), so those two absolute errors carry straight into $\Delta$: worst case $\sim 1.4\times 10^{-9}\text{ m}$. Relative error of $\Delta$ is then $\frac{1.4\times 10^{-9}}{0.25}\approx 5.6\times 10^{-9}$. Why this step? The relative-error amplifier $\frac{|R_1|+|R_2|}{|\Delta |}\approx \frac{1.2756\times 10^7}{0.25}\approx 5.1\times 10^7$ tells us we lose about ${\log }_{10}(5.1\times 10^7)\approx 7.7$ digits. Starting from $\sim 16$ good digits, roughly $8$ remain — so $0.25\text{ m}$ is still fine here. But if the two altitudes were 25 nanometres apart, the amplifier would be $10^{15}$ and nothing would survive. Why this matters: the danger scales with how close the two big numbers are.

3. Engineering fix. Store the difference from a common reference: subtract a base altitude (say $6378137\text{ m}$) from each reading before rounding, or work in a local coordinate frame where the small quantity is represented directly rather than as a difference of two huge ones. Then you keep full precision on the 25 cm quantity, because it is never born from a big-minus-big. Why this step? Never let the interesting small number arise as a difference of two nearly-equal large numbers — represent it directly. See Numerical stability and conditioning.

Verify: $\Delta =0.25$; amplifier $=\frac{R_1+R_2}{\Delta }\approx 5.1\times 10^7$; digits lost $={\log }_{10}(\text{amplifier})\approx 7.7$. ✓

---

Example J — the exam twist

Steps.

1. What float128 buys. `np.float128` (x86 80-bit extended) has $\sim 18$–$19$ significant digits, $u\approx 5.4\times 10^{-20}$. Why this step? Know the new resolution before claiming it helps.

2. Does the cancellation vanish? $\cos (10^{-8})=1-5\times 10^{-17}$; now $5\times 10^{-17}>u$, so the term survives at this particular $x$, and you get $\approx 0.5$. It looks fixed. Why this step? Shows higher precision only postpones the cliff.

3. Push $x$ smaller. At $x=10^{-10}$, $1-\cos x\approx 5\times 10^{-21}<u_{128}$, so float128 also rounds to $0$. The catastrophe returns — you only moved the threshold. Why this step? Proves the problem is the formula's conditioning, not the machine's precision.

4. Correct fix. Reformulate as $\frac{1}{2}{\left(\frac{\sin (x/2)}{x/2}\right)}^2$ (Example D). This works in plain float64 for all small $x$. Why this step? Fixing the expression beats upgrading the hardware — the mistake "more precision always fixes accuracy" from the parent note, cell J.

Verify: at $x=10^{-8}$, float64 naive $=0$ but reformulated $\approx 0.5$; the reformulation needs no extra precision. ✓

---

Recall

Recall Cover the answers — which fix for which cell?

Which cell is "subtract two big same-sign numbers"? ::: A / C — catastrophic cancellation, fix by reformulation. The safe quadratic root uses which sign of $b$? ::: The sign that makes $-b$ add (same sign as $-b\pm \sqrt{\cdot }$), so no cancellation. How do you get the dangerous quadratic root safely? ::: $x_2=c/(a{x}_1)$ from the product of roots. Why does $1-\cos (10^{-8})$ give exactly 0 in float64? ::: The $5\times 10^{-17}$ correction is below $u\approx 1.1\times 10^{-16}$, so $\cos$ rounds to $1$. Best summation order for one-huge-many-tiny? ::: Ascending (smallest first) — or Kahan. Does float128 fix a cancelling formula? ::: No, it only moves the threshold; reformulate instead. What does the Sterbenz lemma guarantee? ::: If two same-sign floats are within a factor of 2, their subtraction is computed exactly (no rounding).