Data augmentation strategies

WHY does augmentation work at all?

HOW: deriving the augmented loss from first principles

Formula From ordinary ERM to augmented ERM

Step 1 — Ordinary Empirical Risk Minimization (ERM). We minimize average loss over the dataset: $\mathcal{L}_{\text{ERM}} = \frac{1}{N}\sum_{i=1}^{N} \ell\big(f_\theta(x_i),\, y_i\big)$ Why this step? This is the baseline: fit the finite samples we happen to have.

Step 2 — Introduce a random transformation $t \sim P(T)$ (e.g. random crop). Because the label is preserved, $(t(x_i), y_i)$ is also a valid training pair. Replace each $x_i$ by its distribution of augmentations: $\mathcal{L}_{\text{aug}} = \frac{1}{N}\sum_{i=1}^{N} \mathbb{E}_{t\sim P(T)}\Big[\ell\big(f_\theta(t(x_i)),\, y_i\big)\Big]$ Why this step? We no longer fit points; we fit neighborhoods around each point.

Step 3 — Monte-Carlo estimate. We can't compute the expectation exactly, so each minibatch samples one transformation per example: $\hat{\mathcal{L}}_{\text{aug}} = \frac{1}{N}\sum_{i=1}^{N} \ell\big(f_\theta(t_i(x_i)),\, y_i\big),\quad t_i\sim P(T)$ Why this step? It's an unbiased estimator of $\mathcal{L}_{\text{aug}}$ , so plain SGD just works — you sample a fresh transform every epoch.

The strategy zoo (with the WHY for each)

Common mistakes (Steel-manned)

Active recall

Recall Feynman: explain to a 12-year-old

Imagine you're learning to recognize your friend from photos. If you only ever saw one photo, you'd get confused when she wears a hat or stands in shadow. So we make lots of copies of the photo — flipped, brighter, zoomed, with a corner covered — and study all of them. Now you recognize your friend anywhere, in any light, even if part of her face is hidden. That's data augmentation: making cheap "trick photos" so the computer isn't fooled by small changes.

Flashcards

What is data augmentation?

Applying label-preserving (or label-mixing) transformations to training data to expand variety and inject desired invariances, acting as regularization.

Write the augmented empirical risk.

\frac{1}{N}\sum_i \mathbb{E}_{t\sim P(T)}[\ell(f_\theta(t(x_i)), y_i)]

Why can SGD use one transform per minibatch?

Because

\ell(f_\theta(t_i(x_i)),y_i)

with

t_i\sim P(T)

is an unbiased Monte-Carlo estimate of the expected augmented loss.

Two mental models for why augmentation works?

(1) More effective training data; (2) injecting invariances = shrinking hypothesis space = regularization.

Give an example where horizontal flip breaks the label.

Text/digits (6↔9), road signs, medical left/right laterality.

Mixup formula for inputs and labels?

\tilde{x}=\lambda x_i+(1-\lambda)x_j

\tilde{y}=\lambda y_i+(1-\lambda)y_j

\lambda\sim\text{Beta}(\alpha,\alpha)

What does CutMix mix the labels by?

Proportional to the pasted patch's area fraction

\lambda

Why does Cutout/Random Erasing help?

It hides part of the object, forcing the model to use the whole object rather than one discriminative patch.

Should you augment the test set?

No (train only), unless deliberately doing test-time augmentation which averages predictions over augments.

Why precompute-once augmentation is worse than on-the-fly?

Precomputing fixes a finite enlarged set; on-the-fly gives fresh transforms every epoch, preventing memorization.

Steel-man: why "more augmentation always better" is wrong?

Too-strong augments push data off the true distribution → underfitting; strength must be tuned via validation.

Mixup label for λ=0.4 between class0 and class1?

[0.4, 0.6]

Connections

Regularization — augmentation is an implicit regularizer.
Overfitting and Generalization — the problem augmentation attacks.
Convolutional Neural Networks — translation invariance pairs with random crops.
Empirical Risk Minimization — augmented ERM extends this.
Batch Normalization & Dropout — complementary regularizers.
Transfer Learning — augmentation especially crucial with small fine-tuning sets.
Beta Distribution — samples the Mixup coefficient $\lambda$ .

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek neural network sirf wahi cheez seekh sakta hai jo usne dekhi hai. Agar aapke paas sirf 10,000 billi ki photos hain, to network wahi exact photos ratt (memorize) leta hai — isko overfitting kehte hain. Data augmentation ka idea simple hai: hum har photo ke thode-thode badle hue versions bana lete hain — flip karke, zoom karke, thodi brightness badal ke, ya ek corner cover karke. Billi ab bhi billi hi hai (label same rehta hai), par network ko har epoch me nayi-nayi photo milti hai, isliye ratt nahi maar sakta — usko asli features (kaan, aankh, shape) seekhne padte hain.

Deep reason ye hai ki hum network ko invariance sikha rahe hain — matlab "ye change hone se answer nahi badalta." Flip hone par bhi cat = cat, isliye network ko rotation/position ke prati invariant banna padta hai. Ye effectively regularization hai: hum function space chhota kar dete hain, isliye generalization achha hota hai. Formula me hum har $x_i$ ko uske transformations ke neighbourhood se replace kar dete hain, aur har minibatch me ek random transform sample karte hain — ye unbiased estimate hota hai, isliye normal SGD bilkul kaam karta hai.

Do zaroori savdhaniyan: (1) Augmentation sirf training set par lagao, test set par nahi — warna galat accuracy milegi. (2) Sirf wahi transform use karo jo aapke domain me sach ho — digits ya text ke liye horizontal flip mat karo, "6" ulta hoke "9" ban jaata hai! Aur advanced tricks jaise Mixup/CutMix me hum do images aur unke labels dono mix karte hain (jaise 70% cat + 30% dog), jisse decision boundary smooth ho jaati hai aur model over-confident nahi hota.

WHY does augmentation work at all?

HOW: deriving the augmented loss from first principles

The strategy zoo (with the WHY for each)

Common mistakes (Steel-manned)

Active recall

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Training Deep Networks

Connections