Sources of error — truncation error, round-off error

1. The two error sources

2. Derive truncation error from scratch (forward difference)

We want $f'(x)$ . Start from Taylor's theorem (the one exact tool):

$f(x+h) = f(x) + h f'(x) + \frac{h^2}{2}f''(x) + \frac{h^3}{6}f'''(x)+\cdots$

Why this step? Taylor expansion is exact (with remainder), so any error we find is honest, not assumed.

Solve for $f'(x)$ :

$f'(x) = \frac{f(x+h)-f(x)}{h} - \underbrace{\frac{h}{2}f''(x) - \frac{h^2}{6}f'''(x)-\cdots}_{\text{what we throw away}}$

Why this step? Rearranging isolates the finite-difference formula and shows precisely the discarded tail.

Why $f''(\xi)$ and not $f''(x)$ ? Taylor's remainder form puts the leftover at some unknown point $\xi\in(x,x+h)$ — exact, no infinite sum needed.

Worked example Central difference is better — derive it

Expand both directions: $f(x+h)=f(x)+hf'+\tfrac{h^2}{2}f''+\tfrac{h^3}{6}f'''+\cdots$ $f(x-h)=f(x)-hf'+\tfrac{h^2}{2}f''-\tfrac{h^3}{6}f'''+\cdots$ Why this step? Subtracting cancels the even-power terms — the $f''$ term vanishes. $f(x+h)-f(x-h)=2hf'+\tfrac{h^3}{3}f'''+\cdots$ $\boxed{f'(x)=\frac{f(x+h)-f(x-h)}{2h}-\frac{h^2}{6}f'''(\xi)}$ Truncation error $=O(h^2)$ — halving $h$ cuts error by 4×, not 2×.

3. Quantify round-off error

When the machine stores $f(x)$ it actually stores $\tilde f(x)=f(x)(1+\delta)$ with $|\delta|\le\varepsilon_{\text{mach}}$ .

For the forward difference, the numerator $f(x+h)-f(x)$ carries an absolute error $\sim \varepsilon_{\text{mach}}\,|f(x)|$ . Dividing by $h$ :

$E_R \;\approx\; \frac{\varepsilon_{\text{mach}}\,|f(x)|}{h}$

Why divide by $h$ ? The error in the numerator is roughly fixed (order $\varepsilon_{\text{mach}}|f|$ ), but we divide by the tiny $h$ , so the relative damage grows as $h\to0$ .

Formula Total error & optimal step

$E_{\text{total}}(h)\approx \underbrace{\frac{h}{2}|f''|}_{\text{truncation}}+\underbrace{\frac{\varepsilon_{\text{mach}}|f|}{h}}_{\text{round-off}}$ Minimise: $\dfrac{dE}{dh}=\dfrac{|f''|}{2}-\dfrac{\varepsilon_{\text{mach}}|f|}{h^2}=0$ $\boxed{h_{\text{opt}}=\sqrt{\frac{2\varepsilon_{\text{mach}}|f|}{|f''|}}\sim\sqrt{\varepsilon_{\text{mach}}}}$ For doubles $\sqrt{\varepsilon_{\text{mach}}}\approx10^{-8}$ — that's the best $h$ for forward differences.

Figure — Sources of error — truncation error, round-off error

4. Worked numerical example

Worked example Differentiate

$f(x)=e^x$ at $x=1$ (true $f'(1)=e=2.71828\ldots$ ) Forward difference $D(h)=\dfrac{e^{1+h}-e^{1}}{h}$ .

$h$	$D(h)$	error
$10^{-1}$	2.8588	$1.4\times10^{-1}$
$10^{-4}$	2.71842	$1.4\times10^{-4}$
$10^{-8}$	2.71828	$\approx10^{-8}$ (best)
$10^{-12}$	2.71827	$\sim10^{-4}$ (round-off blows up!)

Why does $h=10^{-12}$ get worse? Truncation is tiny, but $e^{1+h}$ and $e^1$ agree to ~12 digits; subtracting them loses ~12 digits of precision → round-off dominates. This is the predicted U-shape.

5. Common mistakes

6. Active recall flashcards

What is truncation error?

Error from replacing an exact operation by an approximate/finite one (e.g. cutting a Taylor series); exists even with infinite precision.

What is round-off error?

Error from storing numbers with finitely many digits (finite floating-point precision).

Order of forward-difference truncation error?

O(h)

, equal to

-\frac h2 f''(\xi)

Order of central-difference truncation error?

O(h^2)

, equal to

-\frac{h^2}{6}f'''(\xi)

Why does central difference beat forward?

Symmetric subtraction cancels the

f''

term, leaving

O(h^2)

How does round-off error scale with

h

in forward difference?

\approx \varepsilon_{\text{mach}}|f|/h

, grows as

h\to0

Optimal

h

for forward difference?

h_{\text{opt}}=\sqrt{2\varepsilon_{\text{mach}}|f|/|f''|}\sim\sqrt{\varepsilon_{\text{mach}}}\approx10^{-8}

for doubles.

What is machine epsilon for IEEE double?

\approx 2.22\times10^{-16}

What is catastrophic cancellation?

Loss of significant digits when subtracting two nearly equal numbers.

Test to tell the two errors apart?

Would the error vanish on a perfect-precision calculator? Yes → truncation, No → round-off.

Recall Feynman: explain to a 12-year-old

Imagine measuring a curved hill's steepness. You walk a step forward and divide height gained by step length. If your step is huge, you miss the curve's wiggle — that's truncation error (your "math shortcut" was too rough). If you take a tiny step, the two heights are almost the same, and your ruler can only read so many digits — the tiny difference gets lost in rounding — that's round-off error. So a medium step gives the best answer: not too big, not too small.

Connections

Taylor's Theorem — source of truncation error formulas
Finite Difference Methods — forward/central/backward differences
Floating Point Representation — origin of machine epsilon
Catastrophic Cancellation — round-off amplification
Order of Accuracy — meaning of $O(h^n)$
Newton-Raphson Method — where derivative errors propagate

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, jab bhi computer ya tum khud koi numerical method se answer nikalte ho, woh exact nahi hota. Do main wajah hoti hain. Pehli — truncation error: hum exact process ko beech mein hi kaat dete hain. Jaise derivative ki definition mein limit $h\to0$ hoti hai, lekin practically hum chhota sa finite $h$ le lete hain. Taylor series ka jo tail bachta hai, woh hi truncation error hai. Yeh error perfect calculator pe bhi rahega, kyunki yeh maths ka approximation hai.

Dusri wajah — round-off error: computer har number ko sirf kuch hi digits mein store karta hai (machine epsilon $\approx 10^{-16}$ doubles ke liye). Toh real numbers thoda sa round ho jaate hain. Yeh error tab khatarnaak banta hai jab tum do bahut paas-paas ke numbers ko subtract karte ho — important digits cancel ho jaate hain, sirf noise bachta hai. Isko catastrophic cancellation kehte hain.

Ab maza yahan hai: truncation error chhota hota hai jab $h$ chhota karo, lekin round-off error bada hota hai jab $h$ bahut chhota karo (kyunki $1/h$ bada ho jaata hai). Toh dono opposite directions mein khinchte hain. Best answer ek medium $h$ pe milta hai — jise $h_{opt}\sim\sqrt{\varepsilon_{mach}}\approx10^{-8}$ kehte hain (forward difference ke liye). Isiliye " $h$ jitna chhota, utna accurate" — yeh galat soch hai!

Yaad rakhne ka tarika: Truncate = jaldi trim kiya (bada h mein zyada error), Round = digits khatam (chhota h mein zyada error). Graph banao toh total error ki shape smiley jaisi U hoti hai — bottom pe sweet spot. Exam mein agar forward difference ka order poochhe toh $O(h)$ , central difference ka $O(h^2)$ — kyunki symmetric subtraction se $f''$ term cancel ho jaata hai.

Test yourself — Numerical Methods

Connections