Regression metrics (MAE, MSE, RMSE, MAPE)

Core Concepts

Mean Absolute Error (MAE)

Worked example MAE Calculation

Predicting apartment rent (in $1000s/month):

Sample	Actual ( $y_i$ )	Predicted ( $\hat{y}_i$ )	Error ( $e_i$ ) $\\|e_i\\|$
1	2.5	2.8	-0.3
2	1.8	1.6	0.2
3	3.2	3.9	-0.7
4	2.1	2.0	0.1

$\text{MAE} = \frac{0.3 + 0.2 + 0.7 + 0.1}{4} = \frac{1.3}{4} = 0.325 \text{ (i.e., } \$325\text{)}$

Why this step? Suming absolute errors gives total magnitude; dividing by $n$ converts to per-sample average. Result: "On average, predictions are off by $325."

Root Mean Squared Error (RMSE)

Mean Absolute Percentage Error (MAPE)

Formula MAPE Derivation

Goal: Measure error as a percentage of actual value (scale-independent).

Step 1: Calculate percentage error for each sample. $\text{Percentage Error}_i = \frac{|y_i - \hat{y}_i|}{|y_i|} \times 100\%$

Why divide by $y_i$ ? A $100error on a$ 1000 item (10%) is "worse" in relative terms than $100 on a$ 10,000 item (1%). Step 2: Average the percentage errors. $\text{MAPE} = \frac{100\%}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{|y_i|}$ Units: Percentage (dimensionless). Can compare across different scales (stock prices vs. house prices).

Worked example MAPE Calculation

| Sample | Actual | Predicted | $\|e_i\|$ | $\frac{\|e_i\|}{y_i}$ | Percentage | |--------|--------|----------------------|------------| | 1 | 2.5 | 2.8 | 0.3 | 0.3/2.5 = 0.12 | 12% | | 2 | 1.8 | 1.6 | 0.2/1.8 ≈ 0.111 | 11.1% | | 3 | 3.2 | 3.9 | 0.7 | 0.7/3.2 ≈ 0.219 | 21.9% | | 4 | 2.1 | 2.0 | 0.1/2.1 ≈ 0.048 | 4.8% |

$\text{MAPE} = \frac{12 + 11.1 + 21.9 + 4.8}{4} = \frac{49.8}{4} = 12.45\%$

Why this step? Now sample 3 contributes most (21.9%) not because the absolute error was largest, but because it's 21.9% of the true value. Sample 4 barely registers (4.8%) even though absolute error (0.1) is small.

Interpretation: "On average, predictions are off by 12.45% of the true value."

When to Use Each Metric

Intuition Decision Framework

Use MAE when:

You want interpretability in original units.
All errors should be weighted equally (a $10 error is 2× worse than a$ 5 error, period).
Your data has outliers and you don't want them to dominate (MAE is robust).
Example: Predicting delivery times (30min late is 30min late, whether package was due in 1 hour or 5 hours).

Use MSE when:

You're optimizing during training (gradient descent loves smooth derivatives).
Large errors are disproportionately bad for your application.
You never need to report to non-technical stakeholders (squared units are confusing). Use RMSE when:
You want MSE's sensitivity to outliers BUT interpretable units.
Standard choice for many regression benchmarks (easier to compare papers/models).
Example: Predicting house prices where a $200K miss on a$ 1M house should hurt way more than two $100K misses. Use MAPE when:
Your target variable spans multiple orders of magnitude (stock prices: $5to$ 5000).
Stakeholders think in percentages ("we're 8% off" is clearer than "$230 off").
WARNING: Undefined when $y_i = 0$ . Asymetric (penalizes overestimation more than underestimation).
Example: Sales forecasting across products (comparing error on $10 items vs.$ 1000 items fairly).

Common mistake Common Pitfalls

Mistake 1: "MAE and MSE always rank models the same way." Why it feels right: Both measure error, so lowest error is lowest, right?

The fix: Consider two models on test set:

Model A: Errors = [1, 1, 1] → MAE = 1, MSE = 1
Model B: Errors = [0, 0, 0, 4] → MAE = 1, MSE = 4 MAE says they're tied. MSE says B is 4× worse. Choose metric based on your tolerance for large errors.

Mistake 2: "MAPE is always better because it's scale-independent." Why it feels right: Comparing errors across different-scale datasets seems ideal.

The fix: MAPE has fatal flaws:

Undefined when $y_i = 0$ (division by zero).
Asymetric: Predicting 90when true is 100(10% error) vs. predicting 110 when true is 100 (10% error) → but the first is 10/100, second is 10/110 ≈ 9.1%. Different penalties for same absolute error!
Heavily penalizes underestimation more than overestimation.

Better alternative: Use sMAPE (symmetric MAPE) if you need percentage-based metric: $\text{sMAPE} = \frac{100\%}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2}$

Mistake 3: "I'll just use all metrics and pick the best result."

Why it feels right: More information is better, right?

The fix: This is metric shopping (a form of overfitting). You must choose evaluation metric(s) before training based on business requirements. Otherwise you're cherry-picking results.

Recall Explain to a 12-Year-Old

Imagine you're throwing darts at a target, and we want to measure how good you are.

MAE is like this: After each throw, measure how far the dart is from the bullseye (in centimeters), add them all up, and divide by number of throws. If you're3 cm off, then 5 cm off, then 2 cm off, your average is (3+5+2)/3 = 3.3 cm. Easy!

MSE is different: Instead of just measuring distance, we square it. So 3 cm becomes 9, 5 cm becomes 25, and 2 cm becomes 4. Average is (9+25+4)/3 = 12.7. Notice the 5 cm throw (which was only a bit worse) now counts way more (25 vs. 9). We're punishing big mistakes extra hard!

RMSE is just taking the square root of that12.7, getting 3.6 cm back. We squared to punish big mistakes, then un-squared to get normal units back.

MAPE is for when the target itself changes size. Imagine three dart boards: one is tiny (10 cm across), one is medium (100 cm), one is huge (1000 cm). A5 cm miss on the tiny board is "50% of the board!", but on the huge board it's only "0.5% off." MAPE measures your error as a percentage of the board size, so it's fair across different boards.

Connections

Loss functions in neural networks - MSE is the standard regression loss
Gradient descent - MSE's differentiability makes optimization smooth
Cross-validation - These metrics computed on held-out folds for model selection
Outlier detection - MAE vs. RMSE difference flags outliers
Model comparison - Must fix metric before comparing architectures
Bias-variance tradeoff - High MSE can indicate high variance (overfitting to noise)
Feature scaling - MAPE invariant to scale, MAE/MSE/RMSE affected by target range

#flashcards/ai-ml

What are the four basic regression metrics and their acronyms? :: MAE (Mean Absolute Error), MSE (Mean Squared Error), RMSE (Root Mean Squared Error), MAPE (Mean Absolute Percentage Error)

Write the formula for MAE :: $\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$ where $y_i$ is actual, $\hat{y}_i$ is predicted

What are the units of MAE?

Same units as the target variable (interpretable: dollars, meters, etc.)

Write the formula for MSE

\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

What are the units of MSE and why is this a problem?

Squared units of target (dollars², meters²); not directly interpretable, but mathematically convenient for optimization

Why does MSE penalize large errors more than small errors?

Because errors are squared; a 10-unit error becomes 100 (not 10), while a 2-unit error becomes 4; the ratio is 25:1 instead of 5:1

Write the formula for RMSE

\text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}

What is the relationship between RMSE and MAE? :::≥ MAE always; when RMSE >> MAE, there are large outlier errors; when RMSE ≈ MAE, errors are uniformly distributed

Write the formula for MAPE

\text{MAPE} = \frac{100\%}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{|y_i|}

What are the units of MAPE? :: Percentage (dimensionless); allows comparison across different scales

What is the critical limitation of MAPE?

Undefined when actual value

y_i = 0

(division by zero); also asymmetric (penalizes overestimation differently than underestimation)

When should you use MAE over MSE?

When you want equal weighting of all errors, interpretable units, and robustness to outliers; when errors should scale linearly with magnitude

When should you use RMSE over MAE?

When large errors are disproportionately costly and you want to penalize them more, but still need interpretable units in original scale

When should you use MAPE?

When target variable spans multiple orders of magnitude and relative error matters more than absolute error; when stakeholders think in percentages

Why is MSE preferred during model training even though RMSE is more interpretable?

MSE is differentiable everywhere (smooth gradient), making gradient descent optimization more stable; RMSE's square root introduces numerical instability near zero

What does it mean if RMSE is much larger than MAE?

There are large outlier predictions; the model has some catastrophically wrong predictions that are driving up the squared error

If Model A has errors [2,2,2,2] and Model B has errors [0,0,0,8], calculate MAE and MSE for both. Which metric prefers which model?

Model A: MAE=2, MSE=4; Model B: MAE=2, MSE=16; MAE says tied, MSE strongly prefers A (4 vs 16); demonstrates metric choice matters

What is metric shopping and why is it bad?

Choosing your evaluation metric after seeing results to get the best numbers; it's a form of overfitting/cherry-picking; metric must be chosen before training based on requirements

What is a better alternative to MAPE that fixes its asymetry?

sMAPE (symmetric MAPE):

\text{sMAPE} = \frac{100\%}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2}

Why can't you directly compare MAE values across datasets with different scales?

MAE is in original units; a MAE of 100 means different things for house prices (

100K scale) vs. apartment rents (

1K scale); MAPE would allow fair comparison

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Regression metrics samajhne ke liye ek simple sawal se shuru karte hain: jab apka model continuous values predict kar raha hai (jaise ghar kimat, temperature, ya sales), to apko kaise pata chalega ki model kitna acha hai? Bas yeh dekhna kafi nahi ki "prediction galat hai" - apko yeh measure karna padega ki kitna galat hai aur kis tarah se galat hai.

Chaar main metrics hain jo yeh kaam karte hain. MAE (Mean Absolute Error) sabse seedha hai - har prediction ki galti ko positive number mein convert karo (absolute value), phir average nikalo. Yeh apko bolta hai ki "average mein, apka model itne units se off hai." MSE (Mean Squared Error) thoda different approach leta hai - errors ko square kar deta hai.Iska matlab ki badi galtiyan choti galtiyon se zyada zyada penalize hoti hain.Agar ek jagah 10 units ka error hai aur dosri jagah 2 units ka, to MSE mein 10-unit-wali galti 25 guna zyada contribute karegi (100 vs 4), jabki MAE mein sirf 5 guna (10 vs 2). RMSE bas MSE ka square root hai taki units wapas normal ho jayein - interpretability wapas aa jati hai lekin badi galtiyon ki sensitivity bhi bani rehti hai.

MAPE (Mean Absolute Percentage Error) tab use hota hai jab aapka data bahut alag-alag scales mein hai. Suppose ek product100 rupees ka hai aur dosra 10,000 ka - dono pe50 rupay ka error same nahi hai na? MAPE percentage mein bata hai, to100-wali pe 50% error dikhega aur 10,000-wali pe 0.5%. Lekin dhyan rahe, MAPE tab kaam nahi karta jab actual value zero ho (division by zero), aur yeh asymmetric bhi hai - over-predict aur under-predict ko alag tarike se treat karta hai. Machine learning mein sahi metric choose karna bahut zarori hai kyunki har metric alag chez optimize karta hai - koi outliers ko punish karta hai, koi equal treatment deta hai, koi percentage mein sochta hai. Aapke business requirement ke hisaab se pehle se decide karo, training ke bad nahi!

Regression metrics (MAE, MSE, RMSE, MAPE)

Core Concepts

Mean Absolute Error (MAE)

Mean Squared Error (MSE)

Root Mean Squared Error (RMSE)

Mean Absolute Percentage Error (MAPE)

When to Use Each Metric

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Model Evaluation & Selection

Connections