Universal approximation theorem — Hinglish notes

3.1.7 · AI-ML › Neural Network Fundamentals

WHAT hai yeh theorem?

Ise dhyaan se padho:

Kaunsi class ke functions? Ek compact set par continuous functions (bounded + closed).
Kaunsa architecture? Exactly ek hidden layer jisme width $N$ ho (ek shallow net), linear output.
Kya guarantee hai? Uniform ( $sup$ ) closeness — sirf average par nahi, balki har point par.

WHY sach hai yeh? (Derivation scratch se)

Hum 1-D mein intuition build karte hain, phir general case note karte hain. Goal: dikhao ki squashed sigmoids ke sums koi bhi bump bana sakte hain, aur bumps koi bhi shape bana sakte hain.

Step 1 — Ek single sigmoid ek soft step hai

Sigmoid hai $σ (z) = \frac{1}{1 + e ^{- z}}, σ (- \infty) = 0, σ (+ \infty) = 1.$

Socho $σ (w (x - b))$ . Yeh step kyun? Input ko bade $w$ se multiply karne par transition sharp ho jaata hai; bias $b$ step ki location slide karta hai.

Jab $w \to \infty$ , toh $σ (w (x - b))$ ek hard step ban jaata hai jo $x = b$ par 0 se 1 tak jump karta hai.

Step 2 — Do steps ghataao, bump milega

$b_{1} < b_{2}$ par do steps lo aur ghataao: $bump (x) = σ (w (x - b_{1})) - σ (w (x - b_{2})) .$ Yeh step kyun? Pehla step $b_{1}$ par "on" hota hai; doosra $b_{2}$ par "on" hota hai. Ghataane ka matlab hai hum sirf $b_{1} < x < b_{2}$ ke liye on hain (value $\approx 1$ ), baaki jagah off — ek rectangular bump jisme width $b_{2} - b_{1}$ hai.

Iske liye output weights $+ 1$ aur $- 1$ ke saath 2 hidden neurons chahiye.

Step 3 — Kai bumps scale karo aur rakh do

Domain ko kai chote intervals mein divide karo. Interval $k$ par height $h_{k}$ ka bump rakho: $F (x) = \sum_{k} h_{k} [σ (w (x - b_{k})) - σ (w (x - b_{k + 1}))] .$ Yeh step kyun? Yeh exactly $f$ ka ek staircase / histogram approximation hai. $h_{k} = f (interval k ka midpoint)$ set karo.

Step 4 — Refine karo aur limit lo

Kyunki $f$ ek compact set par continuous hai, woh uniformly continuous hai: interval width ghataao aur har step ki height error $ε$ se neeche aa jaati hai. $w$ bada karne se har step sharp ho jaata hai. Toh staircase uniformly $f$ ki taraf converge karta hai. $■$ (intuitive version)

Figure — Universal approximation theorem

HOW use/interpret karo ise

80/20 core

Common mistakes

Flashcards

Universal Approximation Theorem kya guarantee karta hai?

Ek single-hidden-layer network jisme enough neurons hon, kisi bhi continuous function ko ek compact set par arbitrary accuracy

ε

tak uniformly approximate kar sakta hai.

Theorem ek ___ result hai, efficiency result nahi.

existence.

Kaun se activations shallow net ko universal banate hain (Leshno)?

Koi bhi non-polynomial activation (sigmoid, tanh, ReLU...).

Sigmoids se "bump" kaise banate hain?

Do sharp sigmoid steps ghataao:

σ (w (x - b_{1})) - σ (w (x - b_{2}))

ReLU(x) - ReLU(x-δ) actually kya produce karta hai?

Ek ramp jo

[0, δ]

par rise karta hai aur height

δ

par plateau karta hai — localized bump NAHI; ek sahi bump ke liye aur ReLU units chahiye.

Proof mein compactness kyun matter karta hai?

Yeh uniform continuity deta hai, toh bumps ka itna fine staircase har jagah

ε

ke andar rehta hai.

1-D mein M bumps ke liye neurons (sigmoids)?

N = 2 M

(do sigmoids per bump).

Agar ek layer universal hai toh phir bhi deep networks kyun use karte hain?

Depth wahi functions exponentially kam neurons se achieve karta hai aur train karna aasaan hota hai; akeli width blast ho sakti hai.

Kya theorem acchi generalization ka promise karta hai?

Nahi — sirf ideal weights diye hue domain par fitting; generalization alag hai.

Ek example do jahan theorem apply NAHI hoti.

1/ x

(0, 1]

— domain compact nahi aur function unbounded hai.

Recall Feynman: 12-saal ke bacche ko samjhao

Socho tum koi bhi wiggly line draw karna chahte ho. Tumhare paas ek stamp hai jo ek chhota rectangular block banata hai. Agar tum bahut saare blocks alag-alag heights ke saath side-by-side stack karo, tum koi bhi shape trace kar sakte ho, jaise Lego steps ek curve banate hain. Ek neuron ek "on/off" edge banata hai; do neurons ek block banate hain; bahut saare neurons bahut saare blocks banate hain. Toh enough blocks se tum koi bhi smooth line copy kar sakte ho. Catch yeh hai: kabhi-kabhi tumhe blocks ka bahut bada pile chahiye hoga — isi liye real networks layers stack karte hain taaki smart rahein, instead of millions of blocks pile karne ke.

Connections

Sigmoid activation function — "soft step" bumps supply karta hai.
ReLU activation — non-polynomial, isliye universal bhi.
Depth vs Width tradeoff — kyun deep, wide se efficiency mein better hai.
Uniform continuity — proof ka analysis engine.
Overfitting and generalization — jo theorem cover NAHI karta.
Backpropagation — jisse hum approximating weights (dhundhne ki koshish karte hain) dhundhte hain.