Universal approximation theorem: Worked examples

This is a hands-on companion to the parent theorem note. There we argued why sums of squashed activations can trace any shape. Here we grind through every case by hand — every kind of input the theorem can face — and check each number.

Before we start, one promise: every symbol below is defined the moment it appears.

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The eight examples that follow are each tagged with the cell(s) they cover, so together they touch every box.

#	Case class	What's tricky	Covered by
A	Single sigmoid, generic $x$	reading a step's value at any point	Ex 1
B	Sign of $w$ (positive vs negative)	negative $w$ flips the step	Ex 2
C	Degenerate $w = 0$	the neuron dies — constant output	Ex 3
D	Bump construction (two sigmoids)	subtracting steps to isolate a region	Ex 4
E	Staircase fit of a real $f$	choosing heights, counting neurons	Ex 5
F	Limiting behaviour ( $w \to \infty$ , $ε \to 0$ )	how sharp/how many for target error	Ex 6
G	Assumptions break (non-compact / unbounded)	why no finite net works	Ex 7
H	Exam twist: ReLU instead of sigmoid	ramp $\neq =$ bump; neuron overhead	Ex 8

The figure below shows the three shapes these cells produce, side by side, so you can see at a glance what a "step", a "bump", and a "ramp" each look like — every example rebuilds one of them.

Figure — Universal approximation theorem

Figure s01 — Left (lavender): a single soft step, the object of Cells A and B. Middle (coral): two steps subtracted give a localized bump, Cell D. Right (mint): two ReLUs subtracted give a ramp that never comes back down, Cell H. Notice the bump returns to zero on both sides while the ramp does not.

Worked examples

Worked example Example 1 — Reading a single soft step (Cell A)

Statement. Let one neuron output $s (x) = σ (w (x - b))$ with $w = 1$ , $b = 2$ . Compute $s$ at $x = 2$ , $x = 4$ , $x = 0$ .

Forecast: Guess before computing: what is $s (2)$ ? (Hint: what is $σ (0)$ ?)

At $x = 2$ : the inside is $w (x - b) = 1 \cdot (2 - 2) = 0$ , so $s (2) = σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.5$ . Why this step? The bias $b$ is the step's centre — the place the sigmoid is exactly halfway up. That's why we always locate the transition by setting the inside to $0$ .
At $x = 4$ : inside $= 1 \cdot (4 - 2) = 2$ , so $s (4) = σ (2) = \frac{1}{1 + e ^{- 2}} \approx 0.8808$ . Why this step? To the right of the centre the step has climbed above $\frac{1}{2}$ toward $1$ .
At $x = 0$ : inside $= 1 \cdot (0 - 2) = - 2$ , so $s (0) = σ (- 2) = \frac{1}{1 + e ^{2}} \approx 0.1192$ . Why this step? Left of centre it sits below $\frac{1}{2}$ , heading toward $0$ .

Verify: Symmetry check — $σ (2) + σ (- 2)$ should equal exactly $1$ (the sigmoid is symmetric about $\frac{1}{2}$ ): $0.8808 + 0.1192 = 1.0000$ . ✓

Worked example Example 2 — The sign of

$w$ flips the step (Cell B) Statement. Compare $s_{+} (x) = σ (3 (x - 1))$ and $s_{-} (x) = σ (- 3 (x - 1))$ at $x = 0$ and $x = 2$ .

Forecast: Will $s_{-}$ rise or fall as $x$ increases? Guess before Step 3.

$s_{+} (2)$ : inside $= 3 (2 - 1) = 3$ → $σ (3) \approx 0.9526$ . $s_{+} (0)$ : inside $= 3 (0 - 1) = - 3$ → $σ (- 3) \approx 0.0474$ . So $s_{+}$ rises left-to-right. Why this step? Positive $w$ means "inside" grows with $x$ , dragging the sigmoid up.
$s_{-} (2)$ : inside $= - 3 (2 - 1) = - 3$ → $σ (- 3) \approx 0.0474$ . $s_{-} (0)$ : inside $= - 3 (0 - 1) = 3$ → $σ (3) \approx 0.9526$ . Why this step? Negative $w$ reverses the inside's slope, so the step is now a downward cliff.
Read-off: a negative- $w$ neuron is a mirror image — it starts high and drops. This is how a net makes bumps face either way.

Verify: For any $z$ , $σ (z) + σ (- z) = 1$ . Here $s_{+} (2) + s_{-} (2) = 0.9526 + 0.0474 = 1$ . ✓ The two neurons are exact complements.

Worked example Example 3 — The degenerate neuron

$w = 0$ (Cell C) Statement. A general neuron computes $s (x) = σ (w x + b)$ , where $w$ is its input weight and $b$ is its bias — a constant added inside the sigmoid that slides the whole cliff left or right. Set the weight $w = 0$ and the bias $b = 0.7$ , so $s (x) = σ (0 \cdot x + 0.7) = σ (0.7)$ . Evaluate it.

Forecast: Does the output depend on $x$ at all?

Kill the $x$ -dependence: with $w = 0$ the term $w x$ vanishes, so the input to $σ$ is just the constant bias $b = 0.7$ , no matter what $x$ is. Thus $s (x) = σ (0.7) \approx 0.6682$ for every $x$ . Why this step? A neuron only "sees" the input through $w$ . If $w = 0$ , the input channel is cut and only the bias $b$ survives — the neuron becomes a flat constant.
Consequence: a $w = 0$ neuron contributes no shape, only a constant offset equal to $σ (b)$ . In the staircase it wastes a slot. Why this step? This is the degenerate boundary case: the theorem needs non-constant behaviour, which is exactly what $w = 0$ destroys.

Verify: $σ (0.7) = \frac{1}{1 + e ^{- 0.7}} \approx 0.6682$ , and it must equal $s (999)$ too: $σ (0 \cdot 999 + 0.7) = σ (0.7) = 0.6682$ . ✓ Same everywhere.

Worked example Example 4 — Building a rectangular bump (Cell D)

Statement. Make a bump that is "on" between $b_{1} = 1$ and $b_{2} = 3$ using steepness $w = 10$ : $B (x) = σ (10 (x - 1)) - σ (10 (x - 3)) .$ Evaluate $B$ at $x = 0$ (left of bump), $x = 2$ (inside), $x = 5$ (right).

Forecast: Guess $B (2)$ — should it be near $0$ or near $1$ ?

$x = 2$ (inside): first step $σ (10 \cdot 1) = σ (10) \approx 0.99995$ ; second step $σ (10 \cdot (- 1)) = σ (- 10) \approx 0.0000454$ . So $B (2) \approx 0.99995 - 0.0000454 \approx 0.99991$ . Why this step? Inside the interval the first cliff has already turned on ( $\approx 1$ ) but the second one hasn't yet ( $\approx 0$ ) — on minus off $=$ on.
$x = 0$ (left): both insides negative → $σ (- 10) - σ (- 30) \approx 0.0000454 - 0 \approx 0.0000454 \approx 0$ . Why this step? Left of both cliffs, neither has switched on, so the difference is $\approx 0$ .
$x = 5$ (right): both insides positive → $σ (40) - σ (20) \approx 1 - 1 = 0$ . Why this step? Right of both cliffs, both are on and cancel: on minus on $= 0$ . That's why a bump is localized — off on both sides.

Verify: Three point-checks confirm the shape: $B (0) \approx 0$ , $B (2) \approx 1$ , $B (5) \approx 0$ — near-zero outside the interval and near-one inside, exactly a localized bump. (All three numbers are machine-checked below.) ✓

The next figure draws exactly this construction, so you can see the two dashed cliffs and their solid difference:

Figure s02 — The lavender and butter dashed curves are the two steps at $b_{1} = 1$ and $b_{2} = 3$ ; the coral solid curve is their difference $B$ . Arrows mark that $B$ is $\approx 1$ inside the interval and $\approx 0$ on both sides — a localized rectangular bump.

Worked example Example 5 — Staircase fit of

$f (x) = x^{2}$ on $[0, 1]$ (Cell E) Statement. Approximate $f (x) = x^{2}$ with $4$ bumps. Heights $= f$ at each interval midpoint.

Forecast: How many hidden neurons will $4$ bumps cost?

Split $[0, 1]$ into 4 equal intervals of width $0.25$ : $[0, .25], [.25, .5], [.5, .75], [.75, 1]$ . Midpoints $0.125, 0.375, 0.625, 0.875$ . Why this step? Each bump paints one flat plateau; sampling $f$ at the midpoint balances the error on both sides of the interval.
Heights: $h_{1} = 0.12 5^{2} = 0.015625$ , $h_{2} = 0.37 5^{2} = 0.140625$ , $h_{3} = 0.62 5^{2} = 0.390625$ , $h_{4} = 0.87 5^{2} = 0.765625$ . Why this step? Set each plateau to the true curve height so the staircase hugs the parabola.
Count neurons — derived here, not borrowed. From Example 4 we saw one bump is built from two sigmoid steps (one rising at the left edge, one rising at the right edge). So each bump costs exactly $2$ neurons, and $M$ bumps cost $N = 2 M$ . With $M = 4$ bumps we spend $N = 2 \times 4 = 8$ hidden sigmoids. Why this step? Restating the count from its source (two steps per bump) means you never have to leave this example to know where " $N = 2 M$ " comes from.

Verify: $h_{2} = 0.140625$ and $h_{4} = 0.765625$ ; the four heights are strictly increasing (as $x^{2}$ demands on $[0, 1]$ ), and $N = 8$ . ✓ (checked below)

The staircase and the true parabola are plotted together below:

Figure s03 — Slate curve: the true $f (x) = x^{2}$ . Mint dots: the four midpoint heights $h_{k}$ . Coral flats: each bump's plateau sitting at $h_{k}$ . Lavender dashed: the assembled 8-neuron staircase hugging the parabola.

Worked example Example 6 — Limiting behaviour: how sharp, how many? (Cell F)

Statement. (a) As $w \to \infty$ , what does the step $σ (w (x - b))$ become — including at $x = b$ ? (b) Given the standard midpoint-staircase error bound $ε \leq L \cdot Δ x$ (where $L$ is the largest slope of $f$ and $Δ x = 1/ M$ is the interval width), how many bumps and neurons reach $ε = 0.1$ , and then $ε = 0.0125$ , for $f (x) = x^{2}$ on $[0, 1]$ ?

Forecast: Guess the neuron cost of the finer fit.

(a) The sharp-step limit — all three cases. Fix a point and send $w \to \infty$ :
- $x > b$ : inside $= w (x - b) \to + \infty$ , so $σ \to 1$ .
- $x < b$ : inside $\to - \infty$ , so $σ \to 0$ .
- $x = b$ (the jump point): inside $= w \cdot 0 = 0$ for every $w$ , so $σ (0) = \frac{1}{2}$ always — the limit sits at exactly $\frac{1}{2}$ .
So the soft step becomes a hard step that is $0$ left of $b$ , $1$ right of $b$ , and pinned at $\frac{1}{2}$ on the jump. Why this step? Large $w$ is the "sharpness knob"; naming the $x = b$ value stops you from ever claiming the hard step is "0 or 1 everywhere" — the boundary point is genuinely $\frac{1}{2}$ .
(b) Fix the largest slope $L$ . For $f (x) = x^{2}$ on $[0, 1]$ , the slope is $f^{'} (x) = 2 x$ , largest at $x = 1$ : $L = 2$ . The error bound becomes $ε \leq 2 Δ x = 2/ M$ . Why this step? The staircase can only be as wrong as $f$ can change across one interval; that change is bounded by slope $\times$ width, and Uniform continuity guarantees it shrinks with $Δ x$ .
Solve for $M$ at $ε = 0.1$ . Set $2/ M \leq 0.1 \Rightarrow M \geq 20$ . Take the smallest integer: $M_{0} = 20$ bumps, costing $N_{0} = 2 M_{0} = 40$ neurons. Why this step? We invert the same bound rather than inventing a number, so the reference fit is now consistent with its own inequality.
Refine to $ε = 0.0125$ . Since $ε \leq 2/ M$ means $M \geq 2/ ε$ , driving $ε$ down by the factor $\frac{0.1}{0.0125} = 8$ multiplies $M$ by $8$ : $M = 8 \times 20 = 160$ bumps, i.e. $N = 2 \times 160 = 320$ neurons. Why this step? Error and $1/ M$ move together, so an $8 \times$ smaller error needs an $8 \times$ finer grid — and this is exactly why width can explode for small $ε$ . See Depth vs Width tradeoff.

Verify: $L = 2$ ; $M_{0} = ⌈ 2/0.1 ⌉ = 20$ , $N_{0} = 40$ ; ratio $0.1/0.0125 = 8$ ; $M = 160$ , $N = 320$ . ✓ (checked below)

Worked example Example 7 — When the assumptions break (Cell G)

Statement. A one-hidden-layer network computes $F (x) = \sum_{j = 1}^{N} α_{j} σ (w_{j} x + b_{j})$ , where for each hidden neuron $j$ : $w_{j}$ is its input weight, $b_{j}$ its bias, $σ$ the sigmoid, and $α_{j}$ the output weight — the number that scales that neuron's contribution before everything is summed. Show that with $ε = 1$ no such finite $F$ uniformly approximates $f (x) = \frac{1}{x}$ on $(0, 1]$ .

Forecast: Is the problem the function or the domain — or both?

The output is bounded. Each $σ (\cdot)$ lies strictly in $(0, 1)$ , so $∣ F (x) ∣ = \sum_{j = 1}^{N} α_{j} σ (w_{j} x + b_{j}) \leq \sum_{j = 1}^{N} ∣ α_{j} ∣ =: C$ , a finite constant fixed once the weights are chosen. Why this step? Sigmoids can't exceed $1$ ; a fixed network therefore has a fixed ceiling $C$ on its output magnitude.
The target is not. As $x \to 0^{+}$ , $f (x) = 1/ x \to \infty$ . Pick $x^{⋆} = \frac{1}{C + 2}$ ; then $f (x^{⋆}) = C + 2$ , so $∣ F (x^{⋆}) - f (x^{⋆}) ∣ \geq f (x^{⋆}) - ∣ F (x^{⋆}) ∣ \geq (C + 2) - C = 2 > ε = 1$ . Why this step? Whatever ceiling $C$ the net has, we can find an $x$ where $f$ pokes above it by more than $ε$ .
Diagnosis: the domain $(0, 1]$ is not compact (open at $0$ ) and $f$ is unbounded there. The theorem's hypotheses fail — so its conclusion is not promised. Why this step? This is the boundary/degenerate case for the theorem itself, not for a neuron: assumptions matter.

Verify: With $C = 5$ , $x^{⋆} = \frac{1}{7} \approx 0.1429$ , $f (x^{⋆}) = 7$ , gap $\geq 7 - 5 = 2 > 1$ . ✓ (checked below)

Worked example Example 8 — Exam twist: ReLU makes ramps, not bumps (Cell H)

Statement. ReLU activation is $R (z) = max (0, z)$ . Show that $R (x) - R (x - δ)$ with $δ = 2$ is a ramp that plateaus at $2$ , not a localized bump. Evaluate at $x = - 1, 1, 5$ .

Forecast: Does the output ever return to $0$ on the right side (like a sigmoid bump does)?

$x = - 1$ (far left): $R (- 1) - R (- 3) = 0 - 0 = 0$ . Why this step? Both arguments are negative, so both ReLUs clip to $0$ — a flat floor on the left, same as a bump.
$x = 1$ (mid-rise): $R (1) - R (1 - 2) = R (1) - R (- 1) = 1 - 0 = 1$ . Why this step? Only the first ReLU has switched on ( $R (1) = 1$ ); the second is still clipped ( $R (- 1) = 0$ ). We are climbing the ramp linearly — output equals how far past $0$ we are.
$x = 5$ (far right): $R (5) - R (3) = 5 - 3 = 2$ . Why this step? Now both ReLUs are on and both grow with the same slope $1$ , so their difference locks at the constant gap $δ = 2$ . Crucially the output stays at $2$ and does not fall back to $0$ .
Conclusion — ramp, not bump. Reading the three values $0 \to 1 \to 2$ : the shape rises and then plateaus forever at height $δ = 2$ , whereas Example 4's sigmoid bump returned to $0$ on its right side. So $R (x) - R (x - δ)$ is not a localized bump. To carve a true bump from ReLUs you must add a downward ramp (more units), a constant-factor overhead — but universality still holds. Why this step? Stating the conclusion explicitly is the whole point of the twist: copying the two-sigmoid recipe blindly gives the wrong shape.

Verify: Values $(0, 1, 2)$ are non-decreasing and the right side is the constant $δ = 2$ , confirming a ramp-then-plateau, unlike Ex 4's bump that returned to $0$ . ✓

The final figure contrasts the ReLU ramp with a sigmoid bump on one axis:

Figure s04 — Mint solid: the ReLU difference climbs then plateaus at $δ = 2$ and never returns to zero (annotated). Coral dashed: a sigmoid bump for comparison, which does return to zero on both sides. This is why two ReLUs are not a drop-in replacement for two sigmoids.

Recall Quick self-test on the matrix

Which cell warns you a single neuron can go flat and useless? ::: Cell C — $w = 0$ kills the $x$ -dependence, leaving only $σ (b)$ . Which cell shows why small $ε$ blows up neuron count? ::: Cell F — error $\leq 2/ M$ , so halving error doubles $M$ and $N = 2 M$ . Two sigmoids give a bump; two ReLUs give a ___ ? ::: ramp that plateaus at $δ$ , not a localized bump. Why does $1/ x$ on $(0, 1]$ defeat every finite net? ::: non-compact, unbounded target vs bounded network output.

Related ideas: Overfitting and generalization (fitting $\neq =$ generalizing) and Backpropagation (finding these weights in practice, which the theorem does not promise).