Loss functions - MSE, cross-entropy — AI-ML notes

1. Where do loss functions come from? (Maximum Likelihood)

WHY this matters: MSE and cross-entropy are not two random formulas to memorize — they are the same recipe (NLL) applied to two different noise models (Gaussian vs. Categorical).

2. Mean Squared Error (MSE) — for regression

Formula Deriving MSE from first principles

The Gaussian density: $p(y\mid\hat y)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(y-\hat y)^2}{2\sigma^2}\right)$ Take negative log for one point: $-\log p = \underbrace{\tfrac12\log(2\pi\sigma^2)}_{\text{constant in }\hat y}+\frac{(y-\hat y)^2}{2\sigma^2}$ Constants and the $\tfrac{1}{2\sigma^2}$ scale don't change the argmin, so minimizing NLL ≡ minimizing $\sum(y-\hat y)^2$ . Averaging over $n$ points: $\boxed{\;\text{MSE}=\frac{1}{n}\sum_{i=1}^n (y_i-\hat y_i)^2\;}$

3. Cross-Entropy — for classification

Figure — Loss functions - MSE, cross-entropy

4. Forecast-then-Verify

5. Common mistakes (Steel-manned)

6. Active Recall — #flashcards/ai-ml

Every standard loss is the negative log-likelihood of what?

The data under an assumed noise/probability model.

MSE arises from assuming what noise distribution on targets?

Gaussian (normal) noise,

\varepsilon\sim\mathcal N(0,\sigma^2)

Write the MSE formula.

\frac{1}{n}\sum_i (y_i-\hat y_i)^2

Why do we square the error in MSE?

To make it positive and to penalize large errors more heavily.

Gradient of

\tfrac12(y-\hat y)^2

w.r.t.

\hat y

\hat y - y

(the error itself).

Cross-entropy arises from assuming what distribution?

Bernoulli (binary) / Categorical (multi-class).

Write binary cross-entropy.

-[y\log\hat y+(1-y)\log(1-\hat y)]

Write multi-class cross-entropy with one-hot

y

-\sum_k y_k\log\hat y_k = -\log\hat y_{\text{true}}

Gradient of softmax+CE w.r.t. logit

z_k

\hat y_k - y_k

Why not use MSE for classification?

Sigmoid+MSE gradient saturates (vanishes) when confidently wrong; CE keeps gradient

\hat y-y

BCE when

\hat y=0.5

and

y=1

\ln 2 \approx 0.693

(the coin-flip baseline).

What must the input to CE's log be?

A valid probability in

(0,1)

, so apply softmax/sigmoid first.

Recall Feynman: explain to a 12-year-old

Imagine a dartboard. The loss is how far your dart landed from the bullseye. For guessing a number (like temperature), we measure the distance and square it — landing twice as far is four times as bad (that's MSE). For guessing a category (cat vs dog), we don't measure distance; we ask "how surprised are you by the right answer?" If you said "99% dog" and it was a dog, tiny surprise → tiny loss. If you said "1% dog" and it was a dog, huge surprise → huge loss (that's cross-entropy). Training just means nudging your throws to make the loss shrink.

Connections

Gradient Descent — loss is the surface we descend.
Backpropagation — $\partial L/\partial \hat y$ is the seed of backprop.
Softmax and Sigmoid — produce the probabilities CE consumes.
Maximum Likelihood Estimation — the source of all these losses.
Regularization — extra terms added to the loss.
Logistic Regression — BCE with a single linear layer.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, loss function ka matlab simple hai: model ne prediction ki, aur hum ek single number nikaalte hain jo batata hai "kitna galat ho". Training ka pura kaam yahi hai — is number ko minimize karna, kyunki gradient descent ko differentiate karne ke liye ek scalar chahiye. Bina loss ke koi gradient nahi, koi learning nahi.

Ab do main flavours hain. MSE use hoti hai jab output ek real number ho (jaise price, temperature). Iski root maximum likelihood hai — hum maan lete hain ki target me Gaussian noise hai, aur us assumption ka negative log-likelihood nikaalo to bilkul $(y-\hat y)^2$ nikal aata hai. Square isliye lagate hain taaki error positive rahe aur bade errors ko zyada punish kare. Iska gradient super clean hai: $\hat y - y$ , yaani seedha error hi correction ban jaata hai.

Cross-entropy use hoti hai classification me, jaha output ek probability hota hai (softmax ya sigmoid se). Yaha noise model Bernoulli/Categorical hai. Idea: "sahi answer par tum kitne surprised the?" — surprise $= -\log p$ . Agar tumne sahi class ko 0.99 probability di, loss almost zero; agar 0.01 di (confidently galat), to $-\log 0.01 = 2.3$ , bahut bada loss. Yehi asymmetry network ko fast seekhne par majboor karti hai.

Ek important baat: classification me MSE mat use karo. Sigmoid + MSE ka gradient $\hat y(1-\hat y)$ factor ki wajah se saturate ho jaata hai (zero ho jaata hai) jab model confidently galat ho — learning ruk jaati hai. Cross-entropy woh factor cancel kar deti hai, gradient wahi magic $\hat y - y$ reh jaata hai. Yaad rakhne ka trick: real numbers → Gaussian → MSE; categories → coins → Cross-Entropy. Dono ka gradient end me same beautiful form deta hai.

Loss functions - MSE, cross-entropy

1. Where do loss functions come from? (Maximum Likelihood)

2. Mean Squared Error (MSE) — for regression

3. Cross-Entropy — for classification

4. Forecast-then-Verify

5. Common mistakes (Steel-manned)

6. Active Recall — #flashcards/ai-ml

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Neural Network Fundamentals

Connections