Universal approximation theorem: Foundations — AI-ML deep dive

Before you can read the parent note without tripping, you need to earn every symbol it throws at you. Below, each symbol appears once, with (1) its plain-words meaning, (2) the picture it stands for, and (3) why the theorem cannot do without it. Read top to bottom — each one leans on the one above.

1. The input $x$ and the domain $[0, 1]^{n}$

The little box $[0, 1]$ means "all numbers from 0 to 1, endpoints included". Stacking $n$ of these boxes gives $[0, 1]^{n}$ , the unit cube in $n$ dimensions.

The picture: in 1-D it is just a line segment from 0 to 1. In 2-D it is a filled square. In 3-D a solid cube. Everything the theorem promises happens inside this box.

Why the theorem needs it: the box is closed (it includes its edges) and bounded (it doesn't run off to infinity). Those two words together will later be renamed compact, and compactness is the secret engine of the whole proof.

Figure — Universal approximation theorem

Recall Why "closed and bounded" both matter

Closed ::: the endpoints (like $x = 0$ ) are part of the domain — no sneaky open edge where the function can misbehave. Bounded ::: the domain doesn't stretch to infinity — you can fit it inside a finite ruler.

2. The target function $f$ and "continuous"

Continuous means: the curve has no teleports. If you nudge $x$ a tiny bit, $f (x)$ only moves a tiny bit — you can draw it without lifting your pen.

The picture: a smooth pen-stroke versus a broken one with a sudden jump. The theorem can copy the smooth stroke; it gives up on the one that teleports.

3. The tolerance $ε$ and the $sup$

So $sup_{x} ∣ F (x) - f (x) ∣ < ε$ reads: "even at the single worst point, the copy is off by less than $ε$ ."

The picture: draw a fat ribbon of thickness $ε$ centred on the true curve. The promise is that your copy $F$ stays entirely inside the ribbon — never pokes out anywhere. This "everywhere at once" guarantee is called uniform approximation.

Why the theorem needs it: "close on average" would allow big spikes of error hidden by good behaviour elsewhere. The $sup$ forbids every spike — a much stronger, more useful promise.

Recall sup vs average

Why use sup and not average error? ::: Average could hide a huge error at one point behind tiny errors elsewhere; sup controls the worst point, so the copy is good everywhere.

4. The activation $σ$

The picture: an S-shaped curve that starts flat near $0$ on the far left, rises through $\frac{1}{2}$ at $z = 0$ , and flattens near $1$ on the far right. It is a "soft switch" that goes from off (0) to on (1).

The parent note needs $σ$ to be non-constant, bounded, continuous — meaning it actually moves (non-constant), stays trapped between two heights (bounded, here 0 and 1), and has no jumps (continuous). See Sigmoid activation function for the full profile, and ReLU activation for the modern alternative.

Recall Read the sigmoid limits

$σ (- \infty)$ ::: $0$ (far left, switch off). $σ (+ \infty)$ ::: $1$ (far right, switch on). $σ (0)$ ::: $\frac{1}{2}$ (the midpoint of the rise).

5. Weight $w$ , bias $b$ , and the argument $w^{⊤} x + b$

Big $w$ squeezes the S into a near-vertical step — a sharp switch. This is why the proof can turn a soft sigmoid into a hard step.
$b$ decides where the switch flips. Increasing $b$ moves the flip to a new location.

The picture: in 1-D there is no list, so $w^{⊤} x$ is just $w \cdot x$ — plain multiplication. The $⊤$ only earns its keep in higher dimensions, where it is the machinery that lets a neuron respond to a direction in space.

6. Output weights $α_{j}$ and the sum $\sum_{j = 1}^{N}$

The picture: $N$ separate S-curves, each stretched, slid, and scaled, then stacked on top of each other. Two of them subtracted ( $α = + 1$ and $α = - 1$ ) form one bump; many bumps side by side form a staircase that hugs the target $f$ . $N$ is the width — the neuron count. More neurons → finer staircase → smaller $ε$ (this is the Depth vs Width tradeoff).

Recall The three knobs of one neuron

$w_{j}$ ::: steepness of the switch (how sharp the step is). $b_{j}$ ::: horizontal position of the switch. $α_{j}$ ::: the height/sign of that neuron's output.

7. The hidden concept: uniform continuity

Why the theorem needs it: the proof chops $[0, 1]$ into intervals and puts one flat bump on each. Uniform continuity guarantees a single interval width makes every bump's height error smaller than $ε$ simultaneously — so the whole staircase fits inside the $ε$ -ribbon. Without it, one bad region could ruin the $sup$ . Full story in Uniform continuity.

Prerequisite map

Everything on the left of $I$ must be understood before the parent note 3.1.07 Universal approximation theorem (Hinglish) makes sense. Downstream ideas — how the weights are actually learned (Backpropagation) and whether the fit generalizes (Overfitting and generalization) — are separate stories the theorem does not cover.

Equipment checklist

Test yourself — cover the right side and answer before revealing.

The unit cube

[0, 1]^{n}

in plain words

all points whose

n

coordinates each run from 0 to 1, endpoints included — a line segment, square, or solid cube.

What "compact" means here

closed (includes its edges) AND bounded (fits in a finite ruler).

What "continuous" forbids

sudden jumps/teleports — you can draw

f

without lifting the pen (corners are fine).

What

ε

the small positive error you are allowed to make; you pick it, the theorem meets it.

What

sup_{x} ∣ F - f ∣

measures

the worst (largest) gap between copy and truth over all points

x

Shape of the sigmoid

σ (z) = 1/ (1 + e^{- z})

an S-curve rising from 0 to 1, passing through

\frac{1}{2}

z = 0

Effect of a big weight

w

it steepens the S into a near-vertical sharp step.

Effect of the bias

b

it slides the step left or right to a chosen location.

Meaning of

w^{⊤} x

dot product — multiply matching entries of

w

and

x

and add them into one number.

Meaning of

\sum_{j = 1}^{N} α_{j} (\dots)

stack

N

scaled activations;

α_{j}

sets each one's height/sign.

Why uniform continuity is the engine

one interval width makes every bump's error below

ε

at once, so the staircase fits the ribbon everywhere.