Intuition The ONE core ideaA loss function is a single number that measures how wrong a prediction is, and training a network means shrinking that number. To read the MSE and cross-entropy formulas you need only a handful of building blocks — a prediction, a truth, a way to combine many errors into one number, and the two "mathematical magnifying glasses" (the square and the logarithm) that decide how mistakes get punished.
This page assumes you have seen none of the notation on the parent note the Loss Functions topic . We will earn every symbol before it is used. Read top to bottom; each block leans only on the ones above it.
Everything starts with two numbers.
y y correct answer — the thing reality hands us. If the real house price is 3 lakh, then y = 3
Picture: a fixed dot on a number line, painted as the bullseye of a dartboard.Why we need it: without a correct answer there is nothing to be wrong about .
y ^ ^ "the machine's guess of" . So y ^ guess at the answer.
Picture: a second dot — where your dart actually landed.Why we need it: the whole point of a network is to produce y ^ y
Intuition Error = the gapThe error is simply y − y ^ arrow from your dart to the bullseye. A loss function is just a rule for turning that gap into a single "how bad" number.
We rarely have one prediction — we have many. We need notation for "the i
i y i , y ^ i i counter : i = 1 , 2 , 3 , … y i i y ^ i i
Picture: a numbered row of dartboards — board 1, board 2, board 3.Why: a network is judged on a whole dataset, not one point.
Definition The summation symbol∑ i = 1 ∑ n "plug in i = 1 2 3 n . The number under the sigma (i = 1 starts ; the number on top (n stops .
∑ i = 1 3 a i = a 1 + a 2 + a 3
Picture: a funnel that swallows a list of numbers and pours out their total.Why the topic needs it: the loss over a dataset is one number, so we must collapse many per-example errors into a single sum. n
The raw error y − y ^ + 2 − 2 0 hide two bad guesses.
( a ) 2 = a × a erases the sign (a negative times a negative is positive) and grows fast for big inputs: 0. 5 2 = 0.25 2 2 = 4
Picture: a curved "U" bowl — flat and forgiving near zero, steep and punishing far out (see figure).Why the topic needs it: MSE squares the error so that (a) all errors count as positive, and (b) a mistake twice as large is punished four times as hard.
This "square bowl" is the entire shape of MSE. Its lowest point sits exactly where guess equals truth (y ^ = y
MSE works when the answer is a plain number (a price, a temperature). But for classification — "cat or dog?" — the network instead outputs a probability : a number between 0 and 1 saying how sure it is.
A probability is a number from 0 1 y ^ = 0.9 ∈ ( 0 , 1 )
Picture: a slider bar from 0% on the left to 100% on the right.Why: classification losses need a belief , not a raw number, so the machine's guess is a probability. Producing these valid probabilities is the job of Softmax and Sigmoid .
Common mistake Never feed a raw score to the logarithmA raw network score (a "logit") can be negative or huge — it is not a probability. You must squash it into ( 0 , 1 )
For probabilities we do not use the square. We use the logarithm, because it answers a different question: "how surprised should I be?" Before anything else we must pin down its base .
e exp ln e ≈ 2.718 natural logarithm ln ( x ) e exp ( x ) = e x ln exp undo each other, so ln ( e x ) = x
Picture: exp ln Why: softmax uses exp ln
log ln log always means the natural log ln (base e log ln e
− ln ln negative or zero (since probabilities are ≤ 1 loss to be a positive "badness" score, so we flip the sign: surprise = − ln ( y ^ ) . Now:
Confident and right (y ^ ≈ 1 ≈ 0
Confident and wrong (y ^ ≈ 0 → + ∞
This savage punishment of confident-wrong answers is exactly what cross-entropy is built from.
Three last pieces of notation let the compact formulas on the parent note make sense.
Definition The class indexk K K how many classes there are (e.g. K = 3 k counter over classes : k = 1 , 2 , … , K i y ^ k k y k k
Picture: K k k
Definition One-hot vectorThe truth over K K correct class is 1 0 . For "class 1 out of 3": y = [ 1 , 0 , 0 ] y 1 = 1 y 2 = y 3 = 0
Picture: a row of light switches, exactly one flipped ON.
θ θ all the network's tunable numbers (its weights). Changing θ y ^
Picture: a wall of dials the network can turn.
arg max arg min arg max θ f ( θ ) θ f largest ". arg min smallest . The parent note flips a "make-probability-biggest" problem into a "make-loss-smallest" problem — that sign flip is why maximising likelihood equals minimising loss.
Why: training is literally an arg min Gradient Descent , and the tool that computes which way to turn them is Backpropagation .
Read this map as "which small idea you must own before the arrow's target makes sense" . It restates, in one glance, the exact notation chain of this page: the two dots (y , y ^ − ln θ arg min
prediction y-hat and truth y
error gap y minus y-hat (sec 0)
square makes a bowl (sec 2)
index i and sum sigma (sec 1)
average one over n (sec 1)
probability in zero to one (sec 3)
natural log ln equals surprise (sec 4)
minus ln is badness (sec 4)
one-hot label and index k (sec 5)
argmin find best dials (sec 6)
Test yourself — cover the right side and answer out loud.
What does the hat in y ^ "The machine's guess of" — a prediction.
What is the error between truth and prediction? The gap y − y ^
In ∑ i = 1 n Where the counter i 1 n
How do you turn a sum of n Multiply by n 1
Give two reasons MSE squares the error. To remove the sign, and to punish big errors much harder.
What range must a probability live in? Strictly between 0 and 1.
When ML writes "log Natural log ln e
What is ln ( 1 ) 0
What happens to ln ( x ) x → 0 It plunges to − ∞
Why do we put a minus sign in − ln y ^ Log of a probability is negative; the minus makes loss a positive "badness".
What do k K k 1 K K
In a one-hot label, how many entries are 1? Exactly one — the correct class.
Why does − ∑ k = 1 K y k log y ^ k The zeros in the one-hot label kill every non-true class.
Expand − log [ y ^ y ( 1 − y ^ ) 1 − y ] − [ y log y ^ + ( 1 − y ) log ( 1 − y ^ )]
What does θ All the network's tunable weights (its dials).
What does arg min θ L The dial settings that make the loss smallest.
Once every answer feels easy, you are ready for the parent note Loss functions - MSE, cross-entropy (index 3.1.8) , and the deeper machinery in Maximum Likelihood Estimation and Logistic Regression .