Several traps below hinge on the inequality RMSE≥MAE. Let's see why, not just assert it. The whole difference is one choice: MAE runs each error through a straight line (∣e∣), while RMSE runs it through a bowl-shaped parabola (e2) before averaging and un-squaring.
Reading the figure. The horizontal axis is the raw residual ei=yi−y^i; the vertical axis is the penalty that each metric applies to that residual before averaging. The cyan V-shape is ∣e∣ — the exact quantity MAE averages — and it grows in a straight line as the error grows. The amber curve is e2 — the exact quantity MSE averages — and it is a parabola because squaring a bigger number grows faster and faster. So the two curves are literally the two penalty rules; nothing is metaphorical here.
The formal step (Jensen's inequality). Write xi=∣ei∣ for the absolute errors, and let E[⋅] mean "average over the n samples". Because the squaring function x2 is convex (it curves upward everywhere), Jensen's inequality says the average of the squares is at least the square of the average:
E[x2]≥(E[x])2.
The left side is exactly MSE (average of xi2=ei2); the right side is exactly MAE2 (square of the average of xi=∣ei∣). So MSE≥MAE2. Take the square root of both non-negative sides:
RMSE=MSE≥MAE2=MAE.
Geometrically that is the picture: averaging points on the upward-curving amber bowl lands above the bowl evaluated at the average error. The two sides meet only when every ∣ei∣ is identical, because then there is no spread for the curvature to exaggerate.
MAE and RMSE are always reported in the same units as the target variable.
True — MAE averages ∣ei∣ and RMSE square-roots back out of the squaring, so both land in the target's own units (dollars, metres). MSE alone lives in squared units.
RMSE can be smaller than MAE for some datasets.
False — by Jensen's inequality MSE≥MAE2, so RMSE≥MAE always, with equality only when every error has identical magnitude.
If RMSE equals MAE exactly, all the absolute errors must be equal.
True — equality in E[x2]≥(E[x])2 holds only when there is zero spread in the ∣ei∣ values, i.e. every prediction is off by the same amount.
Minimising MSE and minimising MAE always pick the same "best constant prediction".
False — the constant minimising MSE is the mean of the targets, while the constant minimising MAE is the median; these differ whenever the data is skewed.
A model with lower MAE than a rival is guaranteed to have lower MSE too.
False — concretely, model A has errors [1.5,1.5] giving MAE =1.5, MSE =2.25; model B has errors [0,3] giving a higher MAE =1.5... wait — instead take A =[0,3] (MAE 1.5, MSE 4.5) versus B =[1.4,1.4] (MAE 1.4, MSE 1.96): B has both lower MAE and lower MSE, but swap to A =[2.9,0] (MAE 1.45, MSE 4.205) versus B =[1.5,1.5] (MAE 1.5, MSE 2.25) — now A wins on MAE yet loses on MSE, proving lower MAE does not imply lower MSE.
MAPE is dimensionless, so it lets you fairly compare a house-price model against a temperature model.
False — dimensionless is necessary but not sufficient; each percentage error is divided by that problem's∣yi∣, so a problem whose typical true values are near zero (or spread over a very different range) will inflate MAPE for reasons unrelated to model quality, so the "fair comparison" can be illusory.
Squaring errors in MSE is done mainly to remove negative signs.
False — removing signs is a side effect; the point is the quadratic penalty that makes a 10 error hurt 25× more than a 2 error, not 5× more.
MSE being differentiable everywhere is why gradient descent prefers it to MAE during training.
True — ∣ei∣ has a kink (no clean derivative) at ei=0, whereas (yi−y^i)2 is smooth, giving gradient descent a well-defined slope everywhere.
MAPE treats a prediction of 90 (truth 100) and a prediction of 110 (truth 100) as equally wrong.
True — both have absolute error 10 and the same true value 100 in the denominator, so each scores 10/100=10%; the notorious MAPE asymmetry appears only when you compare across different true values (a fixed absolute error is a bigger percentage of a small yi than of a large one).
"MAPE is undefined only when a prediction y^i equals zero."
Wrong — the denominator is ∣yi∣, the true value, so MAPE blows up when a true value is zero, regardless of the prediction.
"To fix negative residuals I can just drop the absolute value in MAE, since squaring in MSE removes signs anyway."
Wrong — dropping ∣⋅∣ in MAE lets positive and negative errors cancel, giving a near-zero score for a terrible model; the whole purpose of ∣⋅∣ is to stop that cancellation.
"RMSE = mean of the RMSEs, so I can average per-sample RMSE values."
Wrong — you must square, then average, then root; averaging square roots first is not the same operation and understates the outlier penalty (a+b=a+b).
"Since MAE is robust to outliers, I should also train my network by minimising MAE for cleaner gradients."
Wrong — MAE is robust for reporting, but its constant ±1 gradient gives no sense of how far off you are, so optimisation can stall; see loss functions.
"A low MAPE guarantees a low RMSE."
Wrong — they answer different questions; a model can hit small percentages on large yi yet carry a big absolute miss on one huge value, keeping MAPE low but RMSE high.
"Because RMSE and MAE are both in dollars, RMSE minus MAE is meaningless."
Wrong — the gap RMSE−MAE is a legitimate, unit-carrying diagnostic: a large gap signals a few dominant outlier errors, a near-zero gap signals uniform errors.
Why does one catastrophic prediction move MSE far more than MAE?
Because MSE squares each error before averaging, so a large ∣ei∣ grows quadratically and dominates the sum, whereas MAE only sums the magnitudes linearly.
Why is RMSE, not MSE, the usual number quoted in model comparison benchmarks?
RMSE keeps MSE's outlier sensitivity but reports in the target's real units, so it is directly interpretable and comparable to MAE across papers.
Why can MAPE penalise under-prediction and over-prediction unequally?
With true value fixed at yi>0, the ratio ∣yi−y^i∣/∣yi∣ is bounded for under-prediction — as y^i→0 it tops out at 1 (i.e. 100%) — but for over-prediction y^i→∞ it is unbounded; e.g. truth 100: predicting 0 scores 100% at most, while predicting 300 already scores 200%, so over-predictions can be punished far harder.
Why does MAE effectively "vote for the median" while MSE "votes for the mean"?
Minimising summed absolute deviations lands on the middle-ranked point (the median), while minimising summed squared deviations is pulled toward the balance point of mass (the mean), which outliers can drag.
Why should you scale or normalise targets before comparing MSE across two problems?
MSE's magnitude depends on the target's units and range, so a raw MSE on house prices dwarfs one on temperatures for reasons unrelated to model quality — see feature scaling.
Choosing a squared penalty (MSE) tolerates many small biased misses but punishes high-variance blow-ups, so your loss choice quietly encodes how much variance you are willing to trade for bias.
If every prediction is perfect, what do all four metrics read?
All read 0 — MAE, MSE, RMSE become 0 because every ei=0, and MAPE is 0 provided no true value is zero (else it is undefined).
A dataset has one true value yk=0; which metrics survive?
MAE, MSE and RMSE are all fine, but MAPE is undefined because it divides by ∣yk∣=0 — use a symmetric variant like sMAPE or switch metrics.
What is sMAPE (symmetric MAPE) and why does it help?
It replaces the denominator ∣yi∣ with the average of true and predicted magnitude, sMAPE=n100%∑i=1n(∣yi∣+∣y^i∣)/2∣yi−y^i∣, which stays finite unless both yi and y^i are zero and treats over- and under-prediction more evenly.
Does a negative true value break MAPE, and does the sign matter?
The absolute value ∣yi∣ in the denominator strips the sign, so a true value of −100 behaves like +100 for scaling; MAPE is only undefined at yi=0, but negative targets still make the "percentage of true value" reading harder to interpret intuitively.
What happens to MAPE when the true value is tiny but nonzero, say yi=0.001?
Even a small absolute error is divided by a near-zero denominator, producing an enormous percentage that can single-handedly dominate the average — MAPE is unstable near zero.
For a single sample (n=1), how do RMSE and MAE relate?
With one point RMSE =e12=∣e1∣= MAE — the inequality collapses to equality because there is no spread across samples.
If all errors have the same magnitude but alternating signs, does MSE notice the signs?
No — squaring erases the sign, so [+3,−3,+3,−3] and [3,3,3,3] give identical MSE, RMSE and MAE; sign information is deliberately discarded.
Recall Quick self-test
RMSE ≥ MAE, with equality when ::: all absolute errors are equal (zero spread).
MAPE breaks down when ::: a true value yi is zero (or near zero).
MSE is preferred for training because ::: it is smooth/differentiable everywhere, unlike the kink in ∣ei∣.
MAE-minimising constant is the ::: median; MSE-minimising constant is the mean.
