Universal approximation theorem: Question bank

Before we start, one shared vocabulary reminder so no symbol ambush happens:

Compact set = a region that is both bounded (fits inside some finite box) and closed (includes its own edge/boundary points). Example: the segment $[0, 1]$ is compact; the segment $(0, 1]$ is not, because the point $0$ has been cut off.
Uniform approximation = the worst error over the whole domain is small, written $sup_{x} ∣ F (x) - f (x) ∣ < ε$ . The symbol $sup$ ("supremum") just means "the largest the gap ever gets". Small $sup$ = good everywhere at once, not just on average.
$ε$ (epsilon) = your chosen tolerance, a tiny positive number. "Approximate to within $ε$ " means "the error never exceeds this number you picked."

True or false — justify

The theorem lets a network compute any function exactly.

False — it gives approximation to within any chosen

ε > 0

, never exact equality; the error can shrink toward zero but a finite network with finite width leaves a nonzero gap.

A single hidden layer is enough for universality.

True — that is precisely the theorem's claim, but "enough" means existence of some width, and that width may be astronomically large.

If a shallow net is universal, deep nets are pointless.

False — see Depth vs Width tradeoff: depth reaches the same accuracy with exponentially fewer neurons and trains more easily; universality is about capability, not cost.

The theorem guarantees the network will generalize to unseen data.

False — it only promises good fit on the training domain given ideal weights; Overfitting and generalization is a completely separate concern.

Gradient descent is guaranteed to find the universal-approximation weights.

False — the theorem is an existence proof; it never mentions learning, and Backpropagation may land in a bad local region or never reach the constructed weights.

Universality holds for any activation function whatsoever.

False — Leshno's sharpening says the activation must be non-polynomial; a polynomial activation makes the whole network a polynomial, which cannot approximate arbitrary continuous functions.

The theorem applies to the function

f (x) = 1/ x

(0, 1]

False — that domain is not compact (open at

0

) and

f

is unbounded there, so both hypotheses fail and no finite net achieves uniform

ε

-closeness near

0

Making the sigmoid weight

w

larger always improves the approximation.

Partly false — larger

w

sharpens each step toward a hard edge, but you also need more bumps (larger

M

) to reduce plateau-height error; sharpness alone cannot fix a coarse staircase.

Two ReLU units subtract to give a rectangular bump, just like two sigmoids.

False —

ReLU (x) - ReLU (x - δ)

gives a ramp that plateaus at height $δ$ , not a localized bump; a true bump needs extra downward ReLU ramps.

A network that perfectly fits its training points is an instance of the theorem working.

Misleading — fitting finitely many points is interpolation, not uniform approximation of a continuous function everywhere on the domain; the theorem is a stronger, sup-norm statement.

Spot the error

"Because

f

is continuous, the staircase error is small — that's all we need."

Continuity alone is not enough; you need continuity on a compact set, which upgrades to Uniform continuity so that a single interval width works everywhere at once.

"The theorem says width

N = 2 M

neurons, so it's always cheap."

The count

N = 2 M

is per-bump bookkeeping in 1-D;

M

itself can grow without bound (and explodes with input dimension

n

), so "cheap" is exactly what the theorem does not promise.

"Sigmoid is bounded, so Cybenko's proof directly makes ReLU universal too."

ReLU is unbounded, so Cybenko's bounded-activation version does not apply; universality for ReLU comes from the later Leshno non-polynomial criterion instead.

"To make a bump we add two steps:

σ (w (x - b_{1})) + σ (w (x - b_{2}))

Adding gives a double-height staircase, not a bump; you must subtract the second step so the net is "on" only between

b_{1}

and

b_{2}

"Universal approximation means zero test error."

It says nothing about test data — it is about approximating a known target on its domain with the right weights, which you may never learn from limited samples.

"The theorem covers discontinuous target functions if they're bounded."

The hypothesis requires the target to be continuous; a jump discontinuity cannot be uniformly approximated by any finite sum of continuous sigmoids to arbitrary

ε

"Since one layer suffices, the number of layers is the only thing that ever matters."

For a single hidden layer the relevant resource is width (

N

); it is width, not depth, that blows up here — depth only enters when you ask about efficiency.

Why questions

Why does compactness appear in the hypotheses at all?

On a compact set a continuous function is automatically bounded and uniformly continuous, which is exactly what lets one fixed interval width bound the error everywhere simultaneously.

Why do we subtract two sigmoids rather than use one?

One sigmoid is a soft step that stays "on" forever to the right; subtracting a second step turns it back "off", carving out a localized rectangular bump we can place anywhere.

Why must the activation be non-polynomial?

A network built from a polynomial activation is itself a polynomial of bounded degree, and polynomials of fixed degree cannot approximate every continuous function on a compact set — the non-polynomial condition is what breaks this ceiling.

Why is the theorem called an existence result rather than a construction?

It proves suitable weights, biases, and width exist, but gives no bound on how many neurons you need in practice nor any algorithm to find the weights.

Why does depth help if width is already universal?

Composition lets each layer reuse and fold the previous layer's features, so many functions that need exponentially many neurons in one wide layer need only linearly many across several deep layers — an efficiency win, not a capability win.

Why does the Sigmoid activation function make a natural building block for the proof?

Its output is squashed to

(0, 1)

and, when the input weight

w

is large, it approximates a clean

0 \to 1

step, which is the primitive from which bumps and staircases are assembled.

Edge cases

What happens if the target domain is open, like

(0, 1)

It is not compact, so the boundedness/uniform-continuity machinery can fail (e.g.

f

could blow up near the missing endpoint), and the theorem offers no guarantee.

What if we demand exact zero error,

ε = 0

Impossible in general with finite width — the theorem only promises error below every positive

ε

, achieved by a possibly ever-growing network, never a finished zero-error one.

What if the target function is constant, say

f (x) = 5

Trivially approximable (even exactly with a single bias-shifted linear output), so this is the easy corner case where universality is obviously fine.

What happens as the input dimension

n

grows?

The bump-tiling count grows explosively (roughly exponentially in

n

), which is the "curse of dimensionality" reason the required width

N

becomes impractical even though universality still formally holds.

What if the activation is the identity function,

σ (z) = z

The whole network collapses to a single linear map

F (x) = W x + b

, which is polynomial (degree 1) and therefore not universal — it can only ever produce straight-line/affine outputs.

What if two bump edges coincide,

b_{1} = b_{2}

The two sigmoid steps cancel exactly and the bump has zero width, contributing nothing — a degenerate case that simply wastes two neurons.