3.2.13 · D5Training Deep Networks
Question bank — Data augmentation strategies
Every answer side gives real reasoning, not a verdict alone.
Notation you need for this page
Before the traps, three ideas the parent note uses that we will lean on constantly. Read these once and the rest reads smoothly.
True or false — justify
Augmentation adds genuinely new information to the dataset.
False — it adds no new information, only new presentations of existing labels; it encodes an invariance prior (a belief about which changes don't matter), which is regularization, not extra evidence.
Applying augmentation can never hurt validation accuracy.
False — augmentations that push samples off the true data distribution (extreme rotations, unnatural color shifts) make the model waste capacity and can cause underfitting, so validation accuracy drops.
Augmentation is a form of regularization.
True — by forcing the model to give the same answer across transformed inputs, it shrinks the space of functions the model can represent, exactly the effect regularization aims for.
If a transform preserves the label, it is always safe to use.
False in general — safety depends on the domain: a horizontal flip preserves "cat" but flips a "6" into a "9" or swaps left/right lungs, so label-preservation must be checked per dataset, not assumed.
Mixup obeys the label-preserving rule that ordinary augmentation follows.
False — Mixup deliberately breaks it: it blends two images and their labels, producing a soft target like , which is a label-changing augmentation with a different purpose (smoother boundaries).
Precomputing augmented images once and reusing them each epoch is equivalent to on-the-fly augmentation.
False — precomputing yields a fixed enlarged set, so the network sees the same variants repeatedly and can memorize them; on-the-fly sampling gives a fresh every epoch, which is what the Monte-Carlo estimate of relies on.
Test-time augmentation and training-time augmentation are the same technique.
False — training augmentation expands the training distribution; test-time augmentation averages predictions over several augments of each test input to reduce variance, and is applied at inference, not during fitting.
Stronger augmentation should always lower the training loss.
False — aggressive augmentation makes the training task harder (images differ every epoch), so training loss typically stays higher while generalization improves; a high training loss is often the sign it's working.
Cutout helps because it makes images look noisier and noise is good.
False — Cutout masks a contiguous region, forcing the model to use the whole object rather than one lucky patch; the benefit is distributed feature reliance, not generic noise.
Spot the error
"We augment the validation set the same way as training so all data is treated consistently."
The error is augmenting validation/test data — you would then measure accuracy on distorted images that don't match deployment conditions, giving misleading metrics; augment training only.
"Mixup with between a cat and dog should be labeled as pure cat, since the cat is still visible."
The error is forcing a hard label — with interpolation coefficient the target should be , matching the actual 50/50 blend; the point of Mixup is a soft mixture, not a snap to one class.
"To teach translation invariance we crop, then always place the crop dead-center."
The error is centering — a centered crop teaches nothing about position; the crop's top-left corner must be sampled randomly so the object appears at many locations.
"Since a flip is a mirror it never changes the label, so I flip my digit-recognition dataset both ways."
The error is applying flips to digits — horizontal flips corrupt asymmetric digits and vertical flips turn "6" into "9"; the invariance simply isn't true in this domain.
"Augmentation replaces the need for a separate regularizer like Dropout or weight decay."
The error is treating them as substitutes — augmentation regularizes by injecting specific invariances, while Dropout and weight decay constrain the model differently; they are complementary and often combined.
"We sample many transforms per example each step to estimate the expected loss more accurately."
The error is inefficiency, not correctness — one per example per step is already an unbiased estimate of , so plain SGD converges; more samples per step just costs compute for little gain.
"Because CutMix pastes a dog patch onto a cat, the label should stay 100% cat — it's still mostly a cat photo."
The error is ignoring area — the label must mix in proportion to patch area (that area ratio is ), so a visible dog region contributes to the target; keeping it 100% cat gives the model inconsistent supervision.
Why questions
Why does augmentation reduce overfitting, mechanistically?
If the same image looks different every epoch, the model cannot memorize exact pixels; it is forced to learn features that survive the transforms — edges, shapes, semantics — which generalize (see Overfitting and Generalization).
Why must be resampled each epoch rather than fixed?
The training loss is an expectation over ; fixing collapses that average to a single point, so the model memorizes those specific variants, whereas fresh sampling keeps each step an unbiased Monte-Carlo draw whose long-run average is the true expectation.
Why does Mixup improve calibration (reduce over-confidence)?
By training on soft targets like , the model learns to output graded probabilities between classes instead of collapsing to a hard 1.0, encouraging linear behavior across the decision boundary.
Why is in Mixup drawn from a Beta Distribution rather than a uniform one?
Because in controls the mixing strength: pushes the interpolation coefficient toward 0 or 1 (near-original images), gives the uniform case, and larger pulls toward 0.5 (heavier blends) — a single tunable knob a plain uniform can't give.
Why is augmentation described as "fitting neighborhoods instead of points"?
Ordinary ERM fits each isolated sample ; augmentation replaces with the expectation over transformed versions, so the loss covers a whole neighborhood around each data point.
Why can too-aggressive augmentation cause underfitting rather than better generalization?
If transforms distort inputs beyond recognition, the label-preserving assumption breaks and the model receives near-noisy targets, so it cannot fit even the true signal — training loss stays high.
Why does Cutout specifically prevent reliance on a single discriminative patch?
Because the masked rectangle is random, any single "shortcut" region (a cat's ear) will sometimes be hidden, so the model must spread its evidence across the whole object to survive every epoch.
Edge cases
What happens if you set Mixup's ?
The mixed image becomes purely with label — with interpolation coefficient Mixup degenerates to ordinary (unmixed) training on the second sample, so no interpolation occurs.
What is the mixed label in CutMix if the pasted patch covers the entire image?
, so the label becomes 100% the pasted class — the "host" image is fully overwritten, which is degenerate and defeats the purpose of mixing.
For a perfectly rotation-symmetric input (e.g. a centered dot), what does rotation augmentation teach?
Effectively nothing — the transformed image is identical to the original, so the invariance is already satisfied and no new gradient signal is produced.
If your dataset is already huge and diverse, is augmentation still useful?
Often less so — augmentation mainly helps when data is limited or lacks certain variations; with abundant natural variety the invariance prior it injects may be redundant, though mild augmentation rarely hurts.
What if a transform's invariance is true but your test distribution never contains that variation?
You spend model capacity being invariant to something irrelevant; it usually doesn't hurt accuracy but wastes learning effort, so augmentations should target variations that actually appear at deployment.
Does augmentation change the number of parameters in the network?
No — it changes only the data the fixed model sees; capacity is unchanged, but the effective function class the model can fit is narrowed by the invariance constraint.
Active recall
Recall One-line self-check
The single sentence that ties all these traps together? ::: Augmentation adds no new information — it encodes which changes should not change the answer (an invariance prior), and its value and dangers both come from whether that invariance is actually true for your domain and whether you honestly estimate the expected loss .