Visual walkthrough — Universal approximation theorem

We work entirely in one input dimension (one number $x$ going in, one number coming out) because everything visual lives there. The higher-dimensional case is the same trick repeated per direction.

---

Step 0 — The words and symbols we are allowed to use

Before any formula, let us name every piece so nothing appears un-earned.

Everything below is assembled from just these seven names.

---

Step 1 — One sigmoid is a soft step

WHAT. The sigmoid is $\sigma (z)=\underset{\text{squash any }z\text{ into }(0,1)}{\underbrace{\frac{1}{1+e^{-z}}}}.$ Here $z$ is whatever number we hand it, and $e^{-z}$ is the exponential — a tool that grows or shrinks very fast, which is exactly what forces the output to slam toward $0$ or $1$ at the extremes.

WHY the exponential and not, say, a straight line? We want a shape that is $0$ far to the left, $1$ far to the right, and smoothly transitions in between — a "soft switch". A straight line would run off to $\pm \infty$; the exponential in the denominator saturates, pinning the value between $0$ and $1$. Read the endpoints straight off: $\sigma (-\infty )=\frac{1}{1+\infty }=0,\sigma (+\infty )=\frac{1}{1+0}=1.$

PICTURE. The single S-curve below: flat-low on the left, flat-high on the right, one smooth climb in the middle.

---

Step 2 — Weight and bias turn the soft step into a sharp step at a chosen place

WHAT. Feed the sigmoid the combination $w(x-b)$: $\sigma (\underset{\text{steepness}}{\underbrace{w}}(x-\underset{\text{location}}{\underbrace{b}})).$

WHY. Two independent controls appear:

• $b$ is the value of $x$ where the input to $\sigma$ is zero, so $\sigma =\frac{1}{2}$ — the middle of the climb. Changing $b$ slides the step left/right.
• $w$ multiplies the distance $(x-b)$, so a large $w$ means the input to $\sigma$ races from very negative to very positive across a tiny window of $x$ — the climb becomes sharp. As $w\to \infty$ the soft step becomes a hard step: $0$ then instantly $1$ at $x=b$.

PICTURE. Three copies of the same sigmoid: gentle ($w$ small), medium, and near-vertical ($w$ large), all crossing height $\frac{1}{2}$ exactly at $x=b$.

---

Step 3 — Subtract two steps to carve out a single bump

WHAT. Place one sharp step at $b_1$ and another at $b_2$ (with $b_1<b_2$), and subtract: $\text{bump}(x)=\underset{\text{turns ON at }{b}_1}{\underbrace{\sigma (w(x-b_1))}}-\underset{\text{turns ON at }{b}_2}{\underbrace{\sigma (w(x-b_2))}}.$

WHY. Track the value as $x$ sweeps right:

• Left of $b_1$: both steps are $0$, so bump $=0-0=0$. Off.
• Between $b_1$ and $b_2$: first step is $1$, second still $0$, so bump $=1-0=1$. On.
• Right of $b_2$: both are $1$, so bump $=1-1=0$. Off again.

The result is a rectangle that is high only on the window $[b_1,b_2]$. We subtracted because subtraction is what cancels the "on" region we no longer want, leaving a localized block. This costs exactly 2 hidden neurons with output weights $+1$ and $-1$.

PICTURE. The two steps drawn faintly, and their difference — a clean rectangular bump — drawn boldly on top.

Recall Why exactly two, not one?

One step alone stays high forever to the right; it can never come back down. A bump must go up and come back down, and each rise/fall is one sigmoid. ::: So the minimum is 2 sigmoids per bump.

---

Step 4 — Scale the bump to any height with the output weight

WHAT. Multiply the whole bump by $\alpha$: $\alpha \cdot \text{bump}(x)=\underset{\text{height dial}}{\underbrace{\alpha }}\left[\sigma \right(w(x-b_1))-\sigma (w(x-b_2)\left)\right].$

WHY. The raw bump has height $1$. We want to match whatever height the target curve $f$ has over that window. Scaling by $\alpha$ (the output weight) stretches the rectangle to height $\alpha$ — positive $\alpha$ points up, negative $\alpha$ points down (needed where $f$ dips below zero).

PICTURE. The same window, three bumps stacked in the drawing: one at height $1$, one scaled to $\alpha =0.5$, one flipped to $\alpha =-0.7$ (below the axis).

---

Step 5 — Line up many bumps into a staircase

WHAT. Chop the domain $[0,1]$ into $M$ adjacent windows at cut points $b_1<b_2<\cdots <b_{M+1}$. On window $k$ set the height to the target value there, $h_k=f({\text{midpoint}}_k)$. Add all the scaled bumps: $F(x)={\sum }_{k=1}^M{h}_k\left[\underset{\text{one rectangle on window }k}{\underbrace{\sigma (w(x-b_k))-\sigma (w(x-b_{k+1}))}}\right].$

WHY. Adjacent rectangles sit side by side with no gap (each window's right edge is the next window's left edge). Because each is high only on its own window, at any given $x$ essentially one bump is active, contributing its height $h_k$. The sum therefore reads off like a histogram / staircase that traces $f$ step by step. Setting $h_k=f(\text{midpoint})$ makes each stair land on the true curve.

PICTURE. The parabola $f(x)=x^2$ in teal, with an orange staircase of $M=4$ bumps hugging it — each flat plateau touching the curve at its midpoint.

---

Step 6 — Refine: why the staircase must converge

WHAT. Shrink every window width and grow $w$ at the same time. Two errors shrink together:

• Height error — how much $f$ varies within a window.
• Edge error — how "soft" (non-vertical) each step still is.

WHY this is guaranteed and not just hoped-for. $f$ is continuous on the closed, bounded interval $[0,1]$, which makes it uniformly continuous (see Uniform continuity): pick any tolerance $\epsilon$, and there is one window width small enough that $f$ changes by less than $\epsilon$ across every window simultaneously. So the flat top of each stair is never more than $\epsilon$ from the curve. Meanwhile large $w$ squeezes the soft edges into near-vertical jumps, killing the edge error. Together: ${\sup }_{x\in [0,1]}|F(x)-f(x)|<\epsilon .$ The symbol $\sup$ ("supremum") means the worst gap anywhere — not the average. We are promising closeness at every single point.

PICTURE. Two panels: left, a coarse 4-bump staircase with visible gaps; right, a fine 16-bump staircase pressed tight against the same parabola — the gap band shrinking.

---

Step 7 — Edge cases: where the picture breaks

Every case must be covered, including the ones that fail.

Case A — Domain not compact / function unbounded. Take $f(x)=\frac{1}{x}$ on the open interval $(0,1]$. As $x\to 0$ the curve shoots to $+\infty$. No finite pile of bounded bumps can reach infinite height, so no $N$ gives uniform $\epsilon$-closeness. Why it breaks: uniform continuity needed a closed, bounded domain; drop that and the engine stalls.

Case B — Target below the axis. Where $f<0$, just use $\alpha <0$ (Step 4). Downward bumps handle it — nothing new needed. Covered.

Case C — ReLU instead of sigmoid. ReLU gives $\max (0,z)$: a flat $0$ then a rising ramp. Subtracting two ReLUs, $\text{ReLU}(x)-\text{ReLU}(x-\delta )$, does not give a rectangle — it gives a ramp that rises then plateaus at height $\delta$. To carve a true localized bump you add a second downward ramp (more units). Universality survives (ReLU is non-polynomial), but each bump costs a few extra neurons.

PICTURE. Three mini-panels: (A) $1/x$ escaping to infinity with a red "unbounded" arrow; (B) a downward bump for negative $f$; (C) the ReLU ramp-plateau vs. the sigmoid rectangle side by side.

---

The one-picture summary

The whole story on one canvas: a lone sigmoid → sharpened into a step → paired into a bump → scaled → tiled into a staircase → refined until it kisses the target curve within $\epsilon$ everywhere.

Recall Feynman retelling — explain it to a 12-year-old

Imagine you want to draw a smooth hill but you're only allowed to stack LEGO blocks. Each sigmoid is a smooth ramp that goes from floor level up to one block high. If you make the ramp super steep it's basically a stair-step. Put one step going up and, a little further along, one step going down, subtract them, and you've built a single block that sits only where you want it. Now line up lots of blocks of different heights side by side — that's a staircase, and if you make the blocks skinny enough it looks just like the smooth hill from far away. The math promise ("universal approximation") is just: because the hill never jumps suddenly, skinny-enough blocks can copy it to any accuracy you like. The catch is you might need a HUGE number of blocks — which is why real networks stack layers (Depth vs Width tradeoff) instead of just widening one row. And copying the hill you were shown says nothing about hills you haven't seen — that's generalization, a different game, and actually finding the right blocks is yet another one (Backpropagation).

---

Recall

Recall Quick self-check

A single sigmoid $\sigma (w(x-b))$ becomes a hard step at which $x$, and what controls its steepness? ::: Step at $x=b$; steepness controlled by $w$. How many sigmoids for one bump, and what output weights? ::: Two, with weights $+1$ and $-1$. What property of $f$ (from compactness) guarantees the staircase converges? ::: Uniform continuity. Why does $1/x$ on $(0,1]$ escape the theorem? ::: Domain isn't compact and the function is unbounded. Does $\text{ReLU}(x)-\text{ReLU}(x-\delta )$ give a bump? ::: No — a ramp that plateaus at height $\delta$.