Foundations — Data augmentation strategies

Before you can read the parent note Data augmentation strategies without stumbling, you need to own every symbol it throws at you. Below, each symbol is introduced only after the one it depends on. Nothing is used before it is built.

1. What is a training example? $(x,y)$

The little subscript $i$ in $x_i,y_i$ just means "the $i$-th example." If we have $N$ examples, $i$ runs $1,2,\dots ,N$.

2. The label as a list of numbers: one-hot $y=[1,0]$

Computers can't store the word "cat." So a label with, say, 2 possible classes (cat, dog) becomes a short list:

The reason the parent note can later write $\tilde{y}=[0.7,0.3]$ ("70% cat, 30% dog") is that a label is already a list of numbers, so we are allowed to make those numbers fractions. You could not do that with the plain word "cat."

3. The network as a function: $f_{\theta }(x)$

If you've met Convolutional Neural Networks, that is one specific design of the box $f$ that is especially good at images.

4. Measuring "how wrong": the loss $\ell$

Picture a bathroom scale that weighs mistakes: feed in "guessed dog, truth was cat" and it reads a big number.

5. Summing over the dataset: $\sum$ and the average

Divide that sum by $N$ (the number of examples) and you get the average. So

${\mathcal{L}}_{\text{ERM}}=\frac{1}{N}{\sum }_{i=1}^N\ell (f_{\theta }(x_i),y_i)$

reads in plain words: "the average mistake over all $N$ training examples." $\mathcal{L}$ (big script-L) is the total loss we try to make small; the little $\ell$ was one example's loss.

6. Randomness: $t\sim P(T)$ and the expectation $\mathbb{E}$

This is the one genuinely new piece of machinery, so we go slow.

7. Putting it together: the augmented risk

Now every symbol in the parent's headline formula is defined:

Compare with plain ERM: the only new thing is the inner $\mathbb{E}$ — "average over augmentations." That single insertion is the entire mathematical content of data augmentation.

8. Why one random transform per batch is "unbiased": Monte-Carlo

Since we can't compute the infinite average $\mathbb{E}$, each epoch we draw one fresh $t_i$ per image and use its loss. Because $t_i$ was drawn from the correct recipe $P(T)$, its expected value is the quantity we wanted. So ordinary SGD, run over many epochs, silently averages out to ${\mathcal{L}}_{\text{aug}}$.

${\hat{\mathcal{L}}}_{\text{aug}}=\frac{1}{N}{\sum }_{i=1}^N\ell (f_{\theta }(t_i(x_i)),y_i),t_i\sim P(T)$

The little hat $\hat{}$ means "an estimate of," not the exact thing.

9. The Greek letters in the mixing formulas: $\lambda$ and $\alpha$

The tilde $\stackrel{~}{}$ (as in $\tilde{x},\tilde{y}$) just means "the new, augmented version."

10. The two words the whole topic exists for

Prerequisite map

Equipment checklist

Self-test: can you say each in one plain sentence before revealing?