Taylor series approximation

WHAT is a Taylor series?

HOW do we derive it? (from scratch)

We want a polynomial $P(x)=c_0 + c_1(x-a) + c_2(x-a)^2 + \dots$ that agrees with $f$ as closely as possible at $x=a$ .

Step 1 — Match the value. Set $x=a$ : every term with $(x-a)$ vanishes, so $P(a)=c_0$ . We want $P(a)=f(a)$ , so $c_0=f(a)$ . Why this step? The constant term controls the height at $a$ ; nothing else touches it.

Step 2 — Match the slope. Differentiate: $P'(x)=c_1 + 2c_2(x-a) + 3c_3(x-a)^2+\dots$ . Set $x=a$ : $P'(a)=c_1$ . We want $P'(a)=f'(a)$ , so $c_1=f'(a)$ . Why this step? Matching slope makes the line touch $f$ tangentially, not just cross it.

Step 3 — Match the curvature. Differentiate again: $P''(x)=2c_2 + 6c_3(x-a)+\dots$ . Set $x=a$ : $P''(a)=2c_2$ , so $c_2=\dfrac{f''(a)}{2}$ . Why this step? The factor $2$ appears because $(x-a)^2$ differentiated twice gives $2$ . This is the seed of the factorial.

Step 4 — See the pattern. Differentiate $n$ times: the term $c_n(x-a)^n$ becomes $n!\,c_n$ at $x=a$ (all lower terms die, all higher terms still have $(x-a)$ ). So $f^{(n)}(a)=n!\,c_n \;\Rightarrow\; c_n=\frac{f^{(n)}(a)}{n!}.$ Why this step? The $n!$ is not magic — it's exactly the number that $x^n$ spits out when differentiated $n$ times. Dividing by it "undoes" that factor.

Plugging back gives the definition. We derived it by simply demanding all derivatives match.

Worked examples

Worked example 3. Second-order approximation of a loss (ML core)

Let $L(\theta)$ be a loss. Around a point $\theta_0$ with step $\Delta$ : $L(\theta_0+\Delta)\approx L(\theta_0)+\nabla L(\theta_0)^\top\Delta+\tfrac12\,\Delta^\top H\,\Delta$ where $H$ is the Hessian. Why this step? This is the multivariable Taylor series to order 2: value + gradient·step + curvature. Minimizing over $\Delta$ (set derivative to $0$ : $\nabla L + H\Delta=0$ ) gives Newton's step $\Delta=-H^{-1}\nabla L$ . Dropping the $H$ term and taking a small step $-\eta\nabla L$ gives gradient descent. So both optimizers are truncated Taylor series.

Common mistakes

Flashcards

What does the

n

-th Taylor coefficient equal?

\dfrac{f^{(n)}(a)}{n!}

— the

n

-th derivative at

a

divided by

n!

Why is there a factorial in the denominator?

Differentiating

(x-a)^n

exactly

n

times yields

n!

; dividing by it cancels that factor so derivatives match.

What is a Maclaurin series?

A Taylor series expanded around

a=0

Give the Maclaurin series of

e^x

1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\dots

How does the truncation error scale near

a

(x-a)^{k+1}

for an order-

k

polynomial (Lagrange remainder).

Which Taylor order corresponds to gradient descent?

First order:

L(\theta_0+\Delta)\approx L(\theta_0)+\nabla L^\top\Delta

Which order corresponds to Newton's method?

Second order, giving step

\Delta=-H^{-1}\nabla L

Does a Taylor series always equal its function?

No — only within its radius of convergence, and some smooth functions (e.g.

e^{-1/x^2}

) aren't equal to their series at all.

Multivariable 2nd-order Taylor form?

f(x_0+\Delta)\approx f(x_0)+\nabla f^\top\Delta+\tfrac12\Delta^\top H\Delta

Recall Explain to a 12-year-old (Feynman)

Imagine you know exactly where a toy car is, how fast it's going, and how fast it's speeding up — all at one moment. From just those three facts you can guess pretty well where it'll be a split-second later. Taylor series does this for any curve: it uses the height, slope, bend, twist... at one spot to draw a copy of the curve nearby. The more facts (derivatives) you use, the better the copy — but only for spots close to where you started. Far away, your guess falls apart.

Connections

Derivatives and gradients — Taylor is built entirely from derivatives.
Gradient Descent — the 1st-order Taylor truncation.
Newton's Method — the 2nd-order Taylor truncation.
Hessian matrix — the curvature term of multivariable Taylor.
Convex functions — 2nd-order Taylor with $H\succeq 0$ implies convexity.
Approximation error and Big-O — remainder term formalizes accuracy.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Taylor series ka core idea simple hai: koi bhi complicated function agar tum ek point ke paas dekho, to woh ek polynomial jaisa behave karta hai. Us point par function ki height (value), slope (pehla derivative), curvature (doosra derivative) — bas yeh saari cheezein match kar lo, aur tumne function ki ek local copy bana li. Jitne zyada derivatives match karoge (higher order), utni achhi copy — lekin sirf us point ke nazdeek. Door jaake sab galat ho jaata hai.

Factorial kyun aata hai? Kyunki jab tum $(x-a)^n$ ko $n$ baar differentiate karte ho to $n!$ nikalta hai. Us $n!$ ko cancel karne ke liye hi hum divide karte hai. Agar factorial bhool gaye, to $e^x$ ka series $x=1$ par infinity ban jaata — bilkul bekaar. Isliye yaad rakho: "divide by n-factorial or terms go feral".

ML mein yeh direct kaam aata hai. Loss function $L(\theta)$ ka first-order Taylor lo — value plus gradient·step — aur usse tumhe gradient descent milta hai (niche ki taraf chalo). Second-order lo — gradient plus Hessian curvature — aur usse Newton's method milta hai, jahan tum ek parabola fit karke seedha uske bottom par jump karte ho. Matlab jo optimizers tum roz use karte ho, woh actually truncated Taylor series hi hai.

Ek warning: Taylor series hamesha function ke barabar nahi hoti. $\frac{1}{1-x}$ ka series sirf $|x|<1$ mein chalta hai. Aur point sahi choose karna — $\ln x$ ko $x=1$ ke paas expand karo, $x=0$ par nahi (wahan defined hi nahi hai). Yaad rakho: Taylor local hai, global nahi.

WHAT is a Taylor series?

HOW do we derive it? (from scratch)

Worked examples

Common mistakes

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Calculus & Optimization Basics

Connections