1.2.1Calculus & Optimization Basics

Functions, limits, and continuity

1,778 words8 min readdifficulty · medium5 backlinks

1. Functions — the input→output machine

WHY the "exactly one" rule? If one input could give two outputs, the machine is ambiguous — you couldn't predict its behaviour, and calculus (which tracks the output) would break.

WHAT it looks like in ML: A neural network is a giant function f(x;θ)=y^f(\mathbf{x}; \theta) = \hat{y}. Input = pixels, output = class score. The whole game is choosing θ\theta so the machine behaves well.


2. Limits — where is the output heading?

HOW to read ε\varepsilonδ\delta (Feynman version): You (the challenger) say "I bet you can't keep the output within ε\varepsilon of LL." I (defender) reply "Give me any ε\varepsilon, and I'll hand you a δ\delta: stay within δ\delta of cc and I guarantee it." If I can always win, the limit is LL. Note 0<xc0<|x-c| excludes x=cx=c itself — that's the "don't care about the point" clause.

Deriving a limit from scratch (no plugging in)

Claim: limx3(2x+1)=7\displaystyle \lim_{x\to 3}(2x+1)=7. Let's prove it with ε\varepsilonδ\delta.

We want f(x)7<ε|f(x)-7|<\varepsilon, i.e. 2x+17=2x6=2x3<ε|2x+1-7| = |2x-6| = 2|x-3| < \varepsilon. Why this step? We rewrote the output-gap in terms of the input-gap x3|x-3|, because δ\delta controls exactly that.

So x3<ε/2|x-3| < \varepsilon/2. Choose δ=ε/2\delta = \varepsilon/2. Then x3<δ2x3<2δ=ε|x-3|<\delta \Rightarrow 2|x-3| < 2\delta = \varepsilon. ✅ Done — the limit is genuinely 77, built from the definition, not guessed.

One-sided limits & the removable hole

Figure — Functions, limits, and continuity

3. Continuity — heading matches arrival

WHY ML cares: Gradient descent needs ff (loss) continuous and differentiable to have usable slopes. The ReLU max(0,x)\max(0,x) is continuous everywhere (no jump) but has a kink at 00 — continuity is guaranteed, differentiability is not. That's why ReLU trains fine but needs a subgradient at 00.


4. Common Mistakes (steel-manned)


5. Flashcards

What are the two defining properties of a function?
Each input in the domain maps to exactly one output; every domain element is assigned an output.
State the ε\varepsilonδ\delta definition of limxcf(x)=L\lim_{x\to c}f(x)=L.
ε>0δ>0\forall\varepsilon>0\,\exists\delta>0 s.t. 0<xc<δf(x)L<ε0<|x-c|<\delta \Rightarrow |f(x)-L|<\varepsilon.
Why does the limit definition use 0<xc0<|x-c| (strict)?
To exclude the point x=cx=c itself — the limit tracks nearby behaviour, not the value at cc.
What are the three conditions for continuity at cc?
f(c)f(c) exists; limxcf(x)\lim_{x\to c}f(x) exists; they are equal.
What does 0/00/0 tell you to do?
It's indeterminate — simplify (factor/rationalize/L'Hôpital); the limit may still exist.
Is ReLU max(0,x)\max(0,x) continuous at 0? Differentiable at 0?
Continuous yes (no jump); differentiable no (corner).
Name the three discontinuity types.
Removable (hole), jump, infinite.
Compute limx1x21x1\lim_{x\to1}\frac{x^2-1}{x-1}.
Factor to x+1x+1, gives 22.
Vertical line test checks what?
Whether a curve is a function (no vertical line hits it twice).

Recall Feynman: explain to a 12-year-old

Imagine walking toward a doorway. The limit is which spot you're clearly heading toward, even if there's a tiny hole in the floor right at the door — you can still tell where you're going. A function is continuous if when you actually reach the door, you land exactly on the spot you were heading to, with no jump, no hole, no falling off a cliff. In AI, the computer learns by rolling a ball downhill on a curve; if the curve has jumps or cliffs, the ball can't roll smoothly — so we want smooth, continuous curves.


Connections

Concept Map

inputs from

outputs form

checked by

formalized by

clause

matched to value gives

requires no

enables

instance is

optimizes

Function: one input one output

Domain X

Range

Vertical line test

Limit: output heading to L

epsilon-delta definition

Excludes point x=c

Continuity: heading equals arrival

Gradient descent

Neural network f x theta

No jumps or holes

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho ek function ek machine hai: ek number daalo, ek hi number bahar aata hai. Yahi "exactly one output" wala rule important hai — warna prediction possible hi nahi hoga. ML mein poora neural network bhi ek bada function hi hai: input pixels daalo, output class score milta hai.

Limit ka matlab hai — "input jab cc ke paas ja raha hai, to output kahan ki taraf ja raha hai?" Point cc pe kya ho raha hai, usse limit ko koi matlab nahi. Jaise x21x1\frac{x^2-1}{x-1} pe x=1x=1 daalo to 0/00/0 aata hai (undefined), lekin factor karke (x+1)(x+1) ban jaata hai, aur limit 22 nikal aati hai. Yaad rakho: 0/00/0 ka matlab "limit exist nahi karti" nahi hai — yeh sirf ishaara hai ki simplify karo.

Continuity ka matlab hai pen uthaye bina graph banana — no hole, no jump, no cliff. Teen conditions: f(c)f(c) exist kare, limit exist kare, aur dono barabar ho (mnemonic: HEA — Here, Expected, Agree). ML mein yeh isliye zaroori hai kyunki gradient descent ek gend ko slope pe neeche rol karta hai; agar curve mein jump ya cliff ho to gend theek se roll nahi karegi.

Ek important baat: continuous hona differentiable hone se kamzor condition hai. ReLU max(0,x)\max(0,x) har jagah continuous hai lekin 00 pe ek "kink" (corner) hai, isliye wahan slope unique nahi. Isliye ReLU train to hota hai, par 00 pe ek subgradient use karna padta hai. Yeh distinction interview aur real training dono mein kaam aata hai.

Go deeper — visual, from zero

Test yourself — Calculus & Optimization Basics

Connections