Machine epsilon — what it means in practice

WHY does machine epsilon exist?

WHAT is a floating-point number?

HOW do we derive $\varepsilon_{\text{mach}} = 2^{-t}$ ?

Formula Derivation from first principles

Take the number $1.0$ . In binary normalized form: $1.0 = (1.\underbrace{000\ldots0}_{t})_2 \times 2^{0}.$ The mantissa has $t$ fractional bits. The smallest change you can make is to flip the last (least significant) bit from $0$ to $1$ : $(1.\underbrace{000\ldots0}_{t-1}1)_2 \times 2^{0} = 1 + 2^{-t}.$ Why this step? Each fractional bit position $k$ contributes $2^{-k}$ ; the last position is $k=t$ , so the smallest representable increment above $1$ is exactly $2^{-t}$ . Therefore the next number after $1$ is $1 + 2^{-t}$ , and the gap is $\boxed{\varepsilon_{\text{mach}} = 2^{-t}.}$

Figure — Machine epsilon — what it means in practice

Worked examples

Worked example 1 — Why

0.1 + 0.2 != 0.3 on a computer $0.1$ in binary is the repeating fraction $0.0\overline{0011}_2$ — it cannot be stored exactly, so it's rounded with relative error $\le u$ . Same for $0.2$ and $0.3$ . Adding two rounded numbers leaves a residue of order $\varepsilon_{\text{mach}}$ : $0.1 + 0.2 = 0.30000000000000004\ldots$ The observed error is $\approx 4.44\times10^{-17} \approx 0.2\,\varepsilon_{\text{mach}}$ (the gap between consecutive floats near $0.3$ is $\tfrac14\varepsilon_{\text{mach}}\cdot\text{(scale)}$ , so the residue is one such gap). Why this step? The discrepancy is exactly the graininess of the number line near $0.3$ , confirming epsilon's meaning.

Worked example 2 — Computing epsilon yourself

Start with $E = 1.0$ . Repeatedly halve it while $1 + E > 1$ is still true:

E = 1.0
while 1.0 + E > 1.0:
    E = E/2
E = 2*E   # last value that still mattered

The loop stops once $1 + E$ rounds back to $1$ , i.e. when $E$ drops below the gap. The final $E \approx 2.22\times10^{-16}$ . Why this step? We're literally probing the smallest perturbation that $1.0$ can still "feel". That is the definition.

Worked example 4 — Catastrophic cancellation amplifies epsilon

Compute $f(x)=\dfrac{1-\cos x}{x^2}$ near $x=10^{-8}$ . True value $\to 1/2$ . Here $1.0$ is stored exactly, but $\cos x \approx 1 - 5\times10^{-17}$ gets rounded to the nearest float of $1$ , i.e. its tiny deviation from $1$ is lost to relative error $\sim\varepsilon_{\text{mach}}\approx 2\times10^{-16}$ . Subtracting two nearly-equal numbers promotes that rounding error in $\cos x$ into a huge relative error, giving garbage like $0$ or $1.1$ . Fix: rewrite as $\dfrac{2\sin^2(x/2)}{x^2}$ to avoid the subtraction. Why this step? The error does not come from the literal $1.0$ (which is exact); it comes from rounding $\cos x$ . Subtraction of near-equal quantities then divides the surviving result by something tiny, blowing up the relative error.

Forecast-then-Verify

Verify: (1) $2^{-23}\approx1.19\times10^{-7}$ . (2) Near $1$ ; near $1000\approx2^{10}$ the gap is $2^{10}\varepsilon_{\text{mach}}$ — gaps grow with magnitude. (3) No — that's underflow/denormals, a separate limit (smallest normalized $\approx2.2\times10^{-308}$ , smallest subnormal $\approx5\times10^{-324}$ ).

Flashcards

What is machine epsilon?

The gap between

1.0

and the next representable float;

\varepsilon_{\text{mach}}=2^{-t}

, the relative precision of the number system.

Value of

\varepsilon_{\text{mach}}

for IEEE double precision?

2^{-52}\approx 2.22\times10^{-16}

(

t=52

mantissa bits).

Why does

\varepsilon_{\text{mach}}=2^{-t}

Mantissa has

t

fractional bits; smallest increment above

1

is flipping the last bit =

2^{-t}

Difference between

\varepsilon_{\text{mach}}

and the smallest positive float?

Epsilon = relative precision near 1 (~

10^{-16}

); smallest positive float = underflow limit (smallest normalized ~

2.2\times10^{-308}

, smallest subnormal ~

5\times10^{-324}

). Different concepts.

What is unit roundoff

u

Max relative rounding error

=\tfrac12\varepsilon_{\text{mach}}=2^{-t-1}

, since rounding lands within half a gap.

Why is 0.1+0.2 != 0.3?

0.1, 0.2, 0.3 aren't exactly representable in binary; rounding leaves a residue of order

\varepsilon_{\text{mach}}

(observed

\approx4.44\times10^{-17}

Correct way to compare two floats?

|x-y|\le \text{tol}\cdot\max(|x|,|y|)

with

\text{tol}\sim10\,\varepsilon_{\text{mach}}

, never ==.

What is catastrophic cancellation?

Subtracting two nearly-equal numbers promotes tiny rounding-error into large relative error.

Is the literal 1.0 stored exactly?

Yes — 1.0 is exactly representable; in

1-\cos x

the error comes from rounding

\cos x

, not from storing

1.0

How does the gap between floats change with magnitude?

Near a number of size

2^e

the gap is

2^e\varepsilon_{\text{mach}}

— gaps grow with magnitude.

\varepsilon_{\text{mach}}

for single precision (

t=23

2^{-23}\approx1.19\times10^{-7}

Recall Feynman: explain to a 12-year-old

Imagine a ruler that can only show lines very close together near the number 1, but the lines get further apart as numbers get bigger. The computer can only point at a line, never between lines. Machine epsilon is the width of the smallest gap between two lines right next to the number 1. If you ask the computer to mark something thinner than that gap, it just snaps to the nearest line — that tiny snapping error is why $0.1+0.2$ comes out a teeny bit wrong. The computer isn't dumb; it just has a smallest "pencil width", and epsilon is that width.

Connections

Floating-point representation (IEEE-754) — where $t$ , exponent, sign come from
Rounding error and unit roundoff — $u=\tfrac12\varepsilon_{\text{mach}}$
Catastrophic cancellation — how epsilon-noise gets amplified
Condition number — output error $\approx$ condition number $\times\,\varepsilon_{\text{mach}}$
Numerical stability of algorithms — designing to keep error $\sim\varepsilon_{\text{mach}}$
Underflow and overflow — the other limits of the number range

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, computer ke paas memory finite hoti hai, isliye woh har real number ko exactly store nahi kar sakta — usko round karna padta hai. Machine epsilon ka matlab hai: number $1.0$ aur uske just baad wale representable number ke beech ka chhota sa gap. Double precision mein yeh gap $2^{-52}\approx 2.2\times10^{-16}$ hota hai. Yeh basically computer ki "pencil ki motai" hai — isse patli koi cheez woh mark hi nahi kar sakta, bas nearest line pe snap ho jaata hai.

Yeh kyun important hai? Kyunki har calculation mein itna chhota relative error chupa hota hai. Isiliye 0.1 + 0.2 bilkul 0.3 ke barabar nahi aata — thoda sa $0.30000000000000004$ aata hai, error lagbhag $4.44\times10^{-17}$ . Yeh koi bug nahi, yeh epsilon ki wajah se hai. Yaad rakho: epsilon sabse chhota number nahi hai (woh underflow hai — smallest normalized ~ $2.2\times10^{-308}$ , subnormals ke saath ~ $5\times10^{-324}$ tak); epsilon to sabse chhota step hai 1 ke aas-paas.

Ek aur cheez: jaise number bada hota jaata hai, gap bhi badhta jaata hai — $x$ ke paas gap lagbhag $x\cdot\varepsilon_{mach}$ hota hai. Aur jab tum do bahut-paas-paas numbers ko subtract karte ho (catastrophic cancellation), tab yeh chhota sa rounding-error bahut bada relative error ban jaata hai. Dhyaan rakho: $1-\cos x$ mein literal $1.0$ to exactly stored hota hai — error to $\cos x$ ko round karne se aata hai. Isliye formulas ko smartly rewrite karna padta hai.

Practical rule: floats ko kabhi == se compare mat karo. Hamesha |x - y| <= tol * max(|x|,|y|) use karo, jahan tol epsilon ka chhota multiple ho. Bas itna samajh lo to numerical methods ki aadhi galtiyan bach jaayengi.

Test yourself — Numerical Methods

Connections