Exercises — Data augmentation strategies

Label-preserving = the correct answer must not change after the transform.

• (a) Horizontal flip — preserving. A mirror-image cat is still a cat.
• (b) Brightness $+20\%$ — preserving. A brighter cat is still a cat.
• (c) 180° rotation when the task is "which way is up" — NOT preserving. You literally changed the up/down answer.
• (d) Random crop that keeps the cat visible — preserving. Key idea: "label-preserving" is relative to the task, not to the pixels. (c) shows the same rotation can be safe or unsafe depending on what $y$ means.

Model 1 — more data: every transform is a free new training example, so the model effectively sees far more variety. Model 2 — injecting invariances: you tell the network which changes must not change the answer, which shrinks the set of functions it can fit — this is exactly Regularization and fights overfitting.

The top-left corner's row can start at $0,1,2,\dots ,(256-224)=32$, giving $256-224+1=33$ choices. Same for the column. $\text{crops}=33\times 33=33^2=1089.$ Why it helps: the cat now appears at $1089$ different positions, so a CNN cannot cheat by memorizing an absolute location — it learns translation-tolerant features.

The rule is $\tilde{y}=\lambda y_A+(1-\lambda )y_B$. $\tilde{y}=0.4[1,0]+0.6[0,1]=[0.4,0.6].$ Why: the target says "40% cat, 60% dog." The network is pushed toward a calibrated soft output rather than an over-confident hard $1.0$.

Patch area $=80\times 80=6400$. Total area $=200\times 200=40000$. ${\lambda }_{\text{cat}}=1-\frac{6400}{40000}=1-0.16=\mathrm{0.84.}$ So the label is $[0.84,0.16]$ (cat, dog). Why area, not a fixed value? The label should match how much of each object is actually visible. $16\%$ of the frame is dog, so $16\%$ dog supervision.

Definition earned: an estimator $\hat{Q}$ is unbiased for a target $Q$ if $\mathbb{E}[\hat{Q}]=Q$ — on average across all randomness, it lands exactly on the true value (no systematic drift). Target (the thing we wish we could compute for one example $x_i$): $Q_i={\mathbb{E}}_{t\sim P(T)}[\ell (f_{\theta }(t(x_i)),y_i)].$ Estimator we actually use: draw one $t_i\sim P(T)$ and report ${\hat{Q}}_i=\ell (f_{\theta }(t_i(x_i)),y_i)$. Proof: take the expectation of the estimator over the same draw: ${\mathbb{E}}_{t_i}\left[{\hat{Q}}_i\right]={\mathbb{E}}_{t_i}[\ell (f_{\theta }(t_i(x_i)),y_i)]=Q_i.$ The expectation of a single sample from $P(T)$ is the expectation over $P(T)$ — that's the definition of $\mathbb{E}$. Averaging over the $N$ examples keeps it unbiased (expectation of a sum = sum of expectations). Hence plain SGD on the one-sample loss chases the right target. Connects to Empirical Risk Minimization.

If $t_i$ is drawn once and then frozen, every epoch reuses the same ${\hat{Q}}_i=\ell (f_{\theta }(t_i(x_i)),y_i)$. The randomness is gone: across epochs you are no longer averaging over $P(T)$, you are minimizing loss on a fixed enlarged dataset of size $N$ with one arbitrary transform each. Consequences: (1) you lose the per-epoch novelty, so memorization becomes possible again; (2) your effective target quietly changes from $Q_i={\mathbb{E}}_t[\ell ]$ to a single random point $\ell (f_{\theta }(t_i(x_i)),y_i)$, which is a biased proxy for the neighbourhood average. Fix: sample a fresh $t_i$ every epoch (on-the-fly augmentation).

With $a=b=\alpha$: $\mathbb{E}[\lambda ]=\frac{\alpha }{\alpha +\alpha }=\frac{\alpha }{2\alpha }=\frac{1}{2}.$ Interpretation: on average the two images are blended 50/50, regardless of $\alpha$. The parameter $\alpha$ only controls the spread: small $\alpha$ pushes $\lambda$ toward $0$ or $1$ (nearly clean images, mild mixing), large $\alpha$ concentrates near $0.5$ (aggressive half-and-half blends). See the shape in the figure.

• Horizontal flip — REJECT (usually). Left vs right laterality matters in chest imaging; a flip can swap heart/organ sides and mislead. Only use if your task is truly side-invariant.
• Small rotation $\pm 10°$ — ACCEPT. Patients aren't perfectly aligned; mild rotation is realistic and label-preserving.
• Brightness/contrast jitter — ACCEPT. Different machines/exposures produce different intensities; this injects sensor invariance.
• Vertical flip — REJECT. An upside-down chest is off the true data distribution; the network wastes capacity on impossible images.
• Random erasing (small) — ACCEPT with care. Forces use of the whole lung field, not one lucky patch (like Dropout but in pixel space). Keep the square small so it rarely hides the actual lesion. Guiding principle: include a transform iff its invariance is true in the medical domain and the result stays on the real data distribution.

Against the claim — they do different jobs. BatchNorm rescales activations to zero-mean/unit-variance per batch; it does not teach the network that a dark cat and a bright cat are the same class. Brightness jitter varies the input so the network learns features robust to lighting before normalization. BatchNorm stabilizes optimization; jitter injects an invariance. They are complementary: keep both. (If anything, augmentation makes each batch more diverse, which slightly changes BatchNorm's running statistics — usually beneficial.)

We need $\mathbb{E}[s^2]$ where $s$ is uniform over the $101$ integers $0,\dots ,100$. $\mathbb{E}[s^2]=\frac{1}{101}{\sum }_{s=0}^{100}{s}^2.$ Use ${\sum }_{s=0}^n{s}^2=\frac{n(n+1)(2n+1)}{6}$ with $n=100$: ${\sum }_{s=0}^{100}{s}^2=\frac{100\cdot 101\cdot 201}{6}=338350.$ So $\mathbb{E}[s^2]=\frac{338350}{101}=3350$. Then $\mathbb{E}\left[\frac{s^2}{10000}\right]=\frac{3350}{10000}=\mathrm{0.335.}$ Interpretation: on average about 33.5% of the label mass goes to the pasted object under this policy — a strong mixing pressure, useful only if that matches how much foreground you want to swap.

Cross-entropy against a soft target $\tilde{y}=[{\tilde{y}}_0,{\tilde{y}}_1]$ is $\ell =-{\sum }_c{\tilde{y}}_c\log p_c$, which is linear in each ${\tilde{y}}_c$. Substitute $\tilde{y}=\lambda y_i+(1-\lambda )y_j$: $\ell (p,\tilde{y})=-{\sum }_c(\lambda y_{i,c}+(1-\lambda )y_{j,c})\log p_c=\lambda \ell (p,y_i)+(1-\lambda )\ell (p,y_j).$ So the mixed-label loss is a convex combination of the two clean-label losses with weights $\lambda ,1-\lambda$. Taking the expectation over $\lambda \sim \text{Beta}(\alpha ,\alpha )$ (mean $\frac{1}{2}$ from 3.3) shows the network is trained to behave linearly between classes — smoother decision boundaries. This mirrors the augmented-ERM view: we minimize an expectation over a random transform (here the transform is "blend with a random partner at strength $\lambda$"). Same ERM machinery, richer $P(T)$.