Exercises — Regression metrics (MAE, MSE, RMSE, MAPE)

$e_i=y_i-{\hat{y}}_i$. We subtract the prediction from the truth, keeping the sign, then take the size.

Samples 1 and 3 have negative residuals (model guessed too high). Samples 2 and 4 are positive (guessed too low). The absolute value throws the sign away because for MAE we only care how far off, not which direction.

Here $n=4$ (four samples), so we add the four absolute errors and divide by 4: $\text{MAE}=\frac{1}{n}{\sum }_{i=1}^n|y_i-{\hat{y}}_i|=\frac{0.3+0.2+0.7+0.1}{4}=\frac{1.3}{4}=\mathrm{0.325.}$ In real units: on average the model is off by $325/month. (Same units as the target — that is MAE's selling point.)

Square each residual first, because MSE punishes big misses harder: $e_i^2:0.09,0.04,0.49,\mathrm{0.01.}$ Again with $n=4$, average the four squares: $\text{MSE}=\frac{0.09+0.04+0.49+0.01}{4}=\frac{0.63}{4}=\mathrm{0.1575.}$ Those units are "($1000s)² " — awkward. So take the square root to get back to real dollars: $\text{RMSE}=\sqrt{0.1575}\approx 0.3969\Rightarrow \approx \$397.$

Here $n=4$ again. Residuals $y_i-{\hat{y}}_i$: $20-22=-2,25-23=2,30-34=-4,15-15=0.$ Absolute: $[2,2,4,0]$, so $\text{MAE}=\frac{2+2+4+0}{4}=\frac{8}{4}=2.0\text{°C}.$ Squares: $[4,4,16,0]$, sum $=24$, $\text{MSE}=\frac{24}{4}=6.0,\text{RMSE}=\sqrt{6.0}\approx 2.449\text{°C}.$ Notice RMSE (2.449) > MAE (2.0): the single 4° miss is pulling RMSE up.

For each sample, percentage error $=\frac{|e_i|}{|y_i|}\times 100\%$. We divide by the true value so a fixed dollar miss counts more on a cheap apartment.

With $n=4$: $\text{MAPE}=\frac{12.0+11.11+21.875+4.762}{4}=\frac{49.747}{4}\approx 12.44\%.$ Sample 3 dominates — not because its dollar error is biggest (it is, but) mainly because it's the largest fraction of its own true value.

• MAE / RMSE: perfectly fine. The residual is $0-0.5=-0.5$, $|e_5|=0.5$, $e_5^2=0.25$. These just add to the sums.
• MAPE: undefined. The term is $\frac{|e_5|}{|y_5|}=\frac{0.5}{0}$ — division by zero. A single true-zero blows up the entire average.

This is MAPE's fatal flaw (also flagged in the parent note): never use it when the target can be zero. See Outlier detection for spotting such degenerate points early.

With $n=4$ for both models. Model A: $\text{MAE}=\frac{1+1+1+1}{4}=1.$ Squares $[1,1,1,1]$, $\text{MSE}=\frac{4}{4}=1$, $\text{RMSE}=\sqrt{1}=1.$ Model B: $\text{MAE}=\frac{0+0+0+4}{4}=1.$ Squares $[0,0,0,16]$, $\text{MSE}=\frac{16}{4}=4$, $\text{RMSE}=\sqrt{4}=2.$

MAE calls them tied. Total error magnitude is identical (4 units spread out either way). RMSE prefers A. Squaring makes B's single big miss ($4^2=16$) dominate, so RMSE flags B as twice as bad.

The figure below shows this as squared-error bars: Model A's four short bars versus Model B's single towering bar. Equal MAE, wildly unequal MSE.

Interpretation: if one catastrophic error is worse than four small ones for your application, trust RMSE. If all errors are equally tolerable, MAE is the honest score. This is exactly the Model comparison judgment call.

Let $a_i=|e_i|\ge 0$, and let $n$ be the number of samples. Then $\text{MAE}=\frac{1}{n}\sum a_i$ (the mean of the $a_i$) and $\text{RMSE}=\sqrt{\frac{1}{n}\sum a_i^2}$ (the root of the mean of the squares).

First, what is variance? The variance of the list $a_1,\dots ,a_n$ is the average of the squared distances of each value from their mean $\bar{a}=\frac{1}{n}\sum a_i$: $\text{Var}(a)=\frac{1}{n}{\sum }_{i=1}^n(a_i-\bar{a}{)}^2.$ It is a sum of squares divided by $n$; every term $(a_i-\bar{a}{)}^2\ge 0$, and a sum of non-negative things divided by a positive $n$ is non-negative. So $\text{Var}(a)\ge 0$ always — that is not an assumption, it is forced by the squares.

Now expand that same variance algebraically (multiply out the square and split the sum): $\text{Var}(a)=\frac{1}{n}\sum a_i^2-2\bar{a}\cdot \frac{1}{n}\sum a_i+{\bar{a}}^2=\underset{\text{mean of squares}}{\underbrace{\frac{1}{n}\sum a_i^2}}-\underset{(\text{mean}{)}^2}{\underbrace{{\bar{a}}^2}}.$ (We used $\frac{1}{n}\sum a_i=\bar{a}$.) Combining the two facts: since $\text{Var}(a)\ge 0$, $\frac{1}{n}\sum a_i^2\ge {\bar{a}}^2=(\frac{1}{n}\sum a_i{)}^2.$ Take square roots (both sides non-negative): $\text{RMSE}\ge \text{MAE}.$

Equality holds exactly when $\text{Var}(a)=0$, which forces every $(a_i-\bar{a}{)}^2=0$, i.e. all the absolute errors are identical (Model A in 3.1: every $|e_i|=1$ gave $\text{RMSE}=\text{MAE}=1$). The bigger the spread of error sizes, the bigger the gap RMSE $-$ MAE.

Both predictions miss by the same absolute amount, $|e|=10$. But MAPE divides by the true value $y=100$ in both cases: $\text{under: }\frac{|100-90|}{100}=\frac{10}{100}=10\%,\text{over: }\frac{|100-110|}{100}=\frac{10}{100}=10\%.$ So with the standard MAPE definition (divide by $y_i$, the true value), these two are actually equal — 10% each.

The asymmetry the parent note warns about appears when you either (a) divide by the prediction, or (b) compare across different true values, or (c) look at the practical bound: over-prediction error is unbounded (predict $\hat{y}=1000$ for $y=100$ → 900% error), while under-prediction is capped at 100% (best you can under-shoot is $\hat{y}=0$ → 100%). That structural ceiling on under-prediction vs. no ceiling on over-prediction is the real asymmetry, and it quietly nudges models trained on MAPE toward under-predicting. Fix: sMAPE, which divides by $\frac{|y|+|\hat{y}|}{2}$.

Requirements: (1) scale-independent (prices span 3 orders of magnitude), (2) survives $y_i=0$.

• MAE / RMSE: fail requirement (1). A $50 error is huge on a $5 product, trivial on a $5000 one — but MAE/RMSE add raw dollars, so the expensive products dominate the average. Not comparable across scales.
• MAPE: satisfies (1) but fails (2) — division by zero on the zero-demand products.
• sMAPE (symmetric MAPE): satisfies both. It divides by $\frac{|y_i|+|{\hat{y}}_i|}{2}$, so it is percentage-based and only undefined if both actual and predicted are 0 simultaneously (rare, and defensibly treated as 0% error).

$\text{sMAPE}=\frac{100\%}{n}{\sum }_{i=1}^n\frac{|y_i-{\hat{y}}_i|}{(|y_i|+|{\hat{y}}_i|)/2}.$

Recommendation: report sMAPE as the headline cross-product number, and keep RMSE per product as a secondary diagnostic so you can still see which individual products carry large raw errors. Pair with Feature scaling thinking: sMAPE is essentially "scale-normalizing the error the way we scale-normalize features."

Training / optimization: use MSE as the loss. Gradient descent needs a smooth (differentiable-everywhere) function of the predictions. MSE's derivative w.r.t. a prediction is clean: $\frac{\partial }{\partial {\hat{y}}_i}(y_i-{\hat{y}}_i{)}^2=-2(y_i-{\hat{y}}_i),$ a smooth line through zero. MAE's derivative is $\pm 1$ with a kink at 0 (the absolute value is not differentiable there), which makes optimization jitter near the minimum. This connects directly to Loss functions in neural networks.

Reporting: use RMSE (or MAE). Squared dollars mean nothing to a stakeholder; RMSE undoes the square so the number is in real units and keeps MSE's outlier sensitivity.

Why not one metric for both? The two roles have conflicting demands: optimization wants smoothness, reporting wants interpretability. MSE wins smoothness but loses units; RMSE wins units but the square root is a needless monotone wrapper during gradient steps (minimizing MSE and RMSE give the same optimum, so you optimize the cheaper MSE and report the readable RMSE).

Here $n=100$. MAE: $\text{MAE}=\frac{99\times 0.1+1\times 10}{100}=\frac{9.9+10}{100}=\frac{19.9}{100}=\mathrm{0.199.}$ Without the outlier, MAE would be $0.1$. So the outlier roughly doubled MAE (0.1 → 0.199).

RMSE: squares first: $99\times 0.01=0.99$, plus $10^2=100$. $\text{MSE}=\frac{0.99+100}{100}=\frac{100.99}{100}=1.0099,\text{RMSE}=\sqrt{1.0099}\approx \mathrm{1.0049.}$ Without the outlier, RMSE would be $0.1$. So the outlier inflated RMSE from $0.1$ to $\approx 1.005$ — roughly a 10× blow-up.

Which reflects typical performance? 99% of predictions are off by only 0.1. MAE (0.199) stays closer to that typical value; RMSE (1.005) is dragged far above it by one point. If you want to describe the typical prediction, MAE is more honest here. If your goal is to not tolerate that one huge miss, RMSE's alarm is the feature you want. Same connection to Outlier detection and Bias-variance tradeoff (a single high-variance error dominates the squared metric).

No single metric satisfies all three, so report a small dashboard:

1. RMSE (in $) — meets (b): squaring makes a $400k miss on a $3M mansion dwarf several $50k misses, so the metric screams about catastrophic luxury errors. Interpretable units satisfy (a) partly ("we're off by ~$X").
2. MAPE or sMAPE (in %) — meets (c): dividing by the true price lets a $50k error on a $100k home (50%) and a $50k error on a $3M home (1.7%) be compared fairly. Percentages also satisfy (a) — non-technical stakeholders think in "we're 6% off." Use sMAPE if any listing can be near-zero or heavily discounted.
3. MAE (in $) — a robust sanity check of typical error, so you can see whether RMSE is high because of systematic error (RMSE ≈ MAE) or a few disasters (RMSE ≫ MAE, from Exercise 3.2).

Validate all three under Cross-validation so the numbers aren't a lucky split, and stratify the folds by price band so luxury homes aren't accidentally all in one fold. Report: "Typical error (MAE) ≈ $X; outlier-sensitive error (RMSE) ≈ $Y; relative error (sMAPE) ≈ Z%." Three numbers, each meeting one requirement, together telling the full story.

The metric with the same units as the target and equal weight per error is MAE. The metric preferred as a training loss because it is smooth/differentiable is MSE. RMSE equals MAE exactly when all absolute errors are identical (zero spread). The metric undefined when a true value is zero is MAPE. For any dataset, RMSE is always greater than or equal to MAE.