Regression metrics (MAE, MSE, RMSE, MAPE): Visual walkthrough

Step 1 — The residual: one arrow per prediction

WHAT. Suppose we predict something with a number. The real answer is $y$ (say the true rent). Our model's guess is $\overset{y}{^}$ (read "y-hat" — the little hat means estimated). The residual is the leftover gap:

$e_{i} = truth for sample i y_{i} - guess for sample i \overset{y}{^}_{i}$

$i$ is just a label — sample number 1, 2, 3, … up to $n$ (the total count).
$e_{i}$ is the residual: how far off we were on sample $i$ .

WHY. Before we can score a model we must first measure each mistake. The residual is that raw measurement — nothing averaged yet, just one arrow from guess to truth.

PICTURE. In the figure, the true value sits on the diagonal "perfect prediction" line. The coloured arrow is the residual: it points from your guess up or down to the truth. A short arrow = good, a long arrow = bad. Notice the sign: if you guessed too low, the arrow points up and $e_{i}$ is positive; if you guessed too high, it points down and $e_{i}$ is negative.

Step 2 — The problem: arrows cancel

WHAT. The obvious first idea — "just average all the residuals" — fails. Watch:

$\frac{1}{n} \sum_{i = 1}^{n} e_{i} = \frac{( - 0.3 ) + ( 0.2 ) + ( - 0.7 ) + ( 0.1 )}{4} = \frac{- 0.7}{4} = - 0.175$

$i = 1 \sum n$ is the sum sign: "add up the thing on the right for every $i$ from $1$ to $n$ ."
Here it just adds the four raw residuals.

WHY this is broken. A $+ 0.7$ mistake and a $- 0.7$ mistake add to zero, as if we made no error at all. A model that overshoots half the time and undershoots the other half would score a perfect $0$ while being wildly wrong. We need to stop plus and minus from cancelling.

PICTURE. Upward (positive) and downward (negative) arrows partly erase each other on the number line. The green marker (the naive mean) lands near zero even though every single arrow is long. That green dot is the lie we must fix.

Step 3 — Fix #1: fold the arrows with absolute value → MAE

WHAT. The first cure: make every arrow point the same way by taking its length only. That is the absolute value $∣ e_{i} ∣$ — strip the sign, keep the size.

$∣ e_{i} ∣ = ∣ y_{i} - \overset{y}{^}_{i} ∣ \Rightarrow MAE = \frac{1}{n} i = 1 \sum n arrow length ∣ y_{i} - \overset{y}{^}_{i} ∣$

$∣ \cdot ∣$ turns $- 0.3$ into $0.3$ and leaves $0.2$ as $0.2$ . Every arrow now points "up".
$\frac{1}{n}$ averages: total length shared equally over $n$ samples.

WHY absolute value and not something else? It is the gentlest fix: it removes signs without changing sizes. A length of $0.7$ stays $0.7$ . So MAE is exactly the average arrow length, in the same units as the target (dollars, metres, minutes).

PICTURE. All four arrows flipped to point the same direction, laid end to end. Their total is $1.3$ ; split over $4$ samples gives the MAE, drawn as one average-length arrow.

Step 4 — Fix #2: square the arrows → MSE

WHAT. The second cure also kills the sign, but by squaring: $e_{i}^{2} = (y_{i} - \overset{y}{^}_{i})^{2}$ . Squaring a negative gives a positive, so signs vanish — and something new happens.

$MSE = \frac{1}{n} i = 1 \sum n arrow turned into a square (y_{i} - \overset{y}{^}_{i})^{2}$

WHY square instead of absolute value? Look at what squaring does to length: a residual becomes the area of a square whose side is that residual.

side $0.2$ → area $0.04$
side $0.7$ → area $0.49$

The side only grew $3.5 \times$ (from $0.2$ to $0.7$ ), but the area grew $\approx 12 \times$ . Squaring makes big mistakes explode. So MSE punishes large errors far more than small ones — the opposite personality from MAE.

Two bonus reasons: squaring is a smooth curve (no sharp corner at zero like $∣ x ∣$ has), which Gradient descent needs to compute slopes for Loss functions in neural networks.

PICTURE. Each residual is drawn as the side of a real square; the square's shaded area is $e_{i}^{2}$ . The $0.7$ square dwarfs the others — visually screaming "outlier".

Step 5 — Fix the units: square-root back down → RMSE

WHAT. MSE lives in squared units. To read it as money again, take the square root — the operation that undoes squaring:

$RMSE = MSE = \frac{1}{n} i = 1 \sum n (y_{i} - \overset{y}{^}_{i})^{2}$

The shrinks $0.1575$ back to a length: $0.1575 \approx 0.397$ .
Because the squaring happened before the root, RMSE keeps the "big errors hurt more" personality — the root doesn't undo the amplification, it only re-labels the units.

WHY. We wanted both things: MSE's harsh treatment of outliers and MAE's readable units. RMSE is the compromise — same units as the target, but tuned to notice large mistakes.

PICTURE. The average square from Step 4 is "un-folded" back into a single side-length. That side is the RMSE, standing next to the MAE arrow so you can compare heights directly.

Step 6 — Make it scale-free: divide by the truth → MAPE

WHAT. A $100 miss means very different things on a $200 item versus a $10,000 item. To make errors comparable across scales, divide each arrow's length by the size of the truth it belongs to:

$MAPE = \frac{100%}{n} \sum_{i = 1}^{n} error as a fraction of truth \frac{∣ y _{i} - y ^ _{i} ∣}{∣ y _{i} ∣}$

The numerator $∣ y_{i} - \overset{y}{^}_{i} ∣$ is the arrow length (as in MAE).
The denominator $∣ y_{i} ∣$ rescales it: what fraction of the true value did we miss by?
$\times 100%$ turns the fraction into a percentage.

WHY divide by $y_{i}$ ? It answers a relative question. A $0.7$ miss on a rent of $3.2$ is a $21.9%$ error; the same $0.7$ miss on a rent of $10.0$ would be only $7%$ . MAPE cares about proportion, not raw size — perfect when your targets span orders of magnitude (see Feature scaling for the same relative-vs-absolute idea).

PICTURE. Each arrow is shown next to its own true value as a bar; the coloured portion is the fraction eaten by the error. Sample 3's little arrow now looks large because its truth bar is short.

Step 7 — The degenerate cases (where the maths breaks)

WHAT. Every metric has an edge where it stumbles. Cover them all, or you will trip:

Situation	MAE	MSE / RMSE	MAPE
$y_{i} = 0$ (true value is zero)	fine	fine	undefined — divides by $0$
One giant outlier	barely moves	jumps	jumps if truth is small
All residuals equal	RMSE $=$ MAE	—	—
Symmetry of over vs under	symmetric	symmetric	asymmetric

WHY MAPE is asymmetric. Predict $90$ when truth is $100$ : error $= 10/100 = 10%$ . Predict $110$ when truth is $100$ : error $= 10/110 \approx 9.1%$ . Same absolute miss, different penalty — because the denominator changed. MAPE quietly forgives overestimation more than underestimation.

PICTURE. Left panel: an outlier balloons the MSE square while the MAE arrow shrugs. Right panel: two arrows of equal length but unequal MAPE, because they're measured against different truths — the asymmetry made visible.

The one-picture summary

Everything above is one journey: residual → kill the sign → choose your personality → fix the units or the scale. The final figure lays all four metrics on the same four arrows, so you can see them diverge.

Recall Feynman retelling — say it out loud

Imagine you guessed a bunch of house prices. For each house draw an arrow from your guess to the real price. Some arrows point up (you were low), some down (you were high). If you just add the arrows they cancel and you fool yourself into thinking you were perfect — that's the trap in Step 2. So you fold every arrow to point the same way. If you fold by just taking the length and average them, that's MAE, in plain dollars. If instead you turn each arrow into a square and average the squares, one giant mistake blows up huge — that's MSE, which cares a lot about disasters, but now it's in weird dollar-squared units, so you square-root it back to dollars and call it RMSE. Finally, if a $100 miss on a cheap thing should count more than on an expensive thing, you divide each arrow by the true price first and average those fractions — that's MAPE, a percentage. The catches: RMSE is always at least as big as MAE (and much bigger when there's an outlier), and MAPE breaks whenever a true value is zero and unfairly forgives overestimates. That's the whole story — four ways to squeeze a pile of arrows into one honest number.

Recall Quick self-test

Why can't we just average the raw residuals? ::: Positive and negative arrows cancel, hiding real error (Step 2). What does squaring do that absolute value doesn't? ::: It amplifies large errors far more than small ones (area grows faster than side). Why take the square root in RMSE? ::: To undo the squared units and return to the original scale (dollars, not dollars²). When is MAPE undefined? ::: When any true value $y_{i} = 0$ (division by zero). RMSE vs MAE — which is larger and what does a big gap mean? ::: RMSE $\geq$ MAE always; a large gap signals outlier errors.

See also: Model comparison · Cross-validation · Bias-variance tradeoff · Hinglish version