5.6.3 · D5Machine Learning (Aerospace Applications)
Question bank — Regularization — L1 (lasso), L2 (ridge), dropout
Before we start, one shared vocabulary reminder so every reveal reads cleanly:
The L1-versus-L2 geometry gets asked below, so ground it once here with a picture:
The blue diamond is the L1 region ; the yellow circle is the L2 region . The red ellipses are loss contours shrinking toward the unconstrained best fit. Notice the diamond is first touched at its sharp corner on the -axis, where exactly — that is a sparse solution. The smooth circle is touched on its side, at a point where both weights are nonzero. Keep this image in mind for the "Why questions" section.
True or false — justify
L2 regularization drives many weights to exactly zero, giving a sparse model.
False — L2's shrinkage is proportional (), so weights approach zero but essentially never land on it. It is L1 that produces exact zeros.
Setting very large always improves generalization.
False — as every weight is crushed toward zero and the model underfits, ignoring the real signal. There is a sweet spot found by Cross-Validation, not a "more is better" rule.
Dropout is applied at test time just like at training time.
False — at test time you keep every neuron and (with inverted dropout) do no scaling; you want the network's full capacity for the final prediction, not a crippled sub-network.
L1 and L2 penalties both keep weights small, so they are interchangeable.
False — they shrink differently: L2 penalizes the square (huge weights punished hardest, tiny ones barely touched), while L1 penalizes the absolute value (constant push that zeroes out small weights). L1 selects features; L2 just smooths.
Regularization reduces training accuracy but that is a bug to be avoided.
False — a slight drop in training accuracy is the intended trade: you sacrifice fitting noise so the model generalizes to new flights. Lower train accuracy with higher test accuracy is a success.
The in the L2 penalty changes the model's behaviour.
False — it is a cosmetic convenience that cancels the factor of 2 from differentiating . You could drop it and just rescale ; the family of achievable models is identical.
With inverted dropout, the expected activation of a neuron during training equals its clean activation.
True — with when kept (probability ) and when dropped (probability ), the average is , which is exactly why no rescaling is needed at test time.
L2 regularization is the same thing as "weight decay."
True (for plain SGD) — rearranging the L2 update gives , and that factor literally decays the weight each step before the data gradient is applied.
Dropout can be used on the input layer with the same that hidden layers use.
False in practice — dropping half the raw inputs destroys too much information; input dropout, if used at all, uses a much smaller (e.g. 0.1–0.2). The rule of thumb is for hidden/FC layers.
Spot the error
"To regularize, I added to the loss and also to the bias term ."
The error is regularizing the bias. The bias just shifts the output level; penalizing it fights the model's ability to center predictions and gives no overfitting benefit. Penalize weights, leave free.
"L1 update: ."
That is the L2 update. L1 subtracts a constant , not a proportional factor: .
" is differentiable everywhere, so L1 gradients are always well-defined."
Wrong — has a kink at where the derivative is undefined. We patch this with a subgradient (any value in ), and choosing there is exactly what lets weights sit at zero.
"I used dropout at test time to get uncertainty estimates, so I scaled activations by instead of ."
Mixing conventions. If you already scaled by during training (inverted dropout), test time needs no scaling at all. Applying it again double-counts and inflates activations.
"Since dropout randomly zeroes neurons, it makes the network a random guesser and hurts training."
The randomness is the point, not a flaw — each pass trains a different sub-network, and the ensemble of ~ sub-networks averages into a robust model (see Ensemble Methods). Loss is noisier but generalization improves.
"My Ridge model still overfits, so I switched the sign to to penalize harder."
A negative penalty rewards large weights — the opposite of regularization. The loss can be driven arbitrarily low by blowing up weights, wrecking the model.
Why questions
Why does the L1 diamond constraint produce sparse solutions while the L2 circle does not?
Look at the figure at the top: the diamond has sharp corners lying on the axes, where one coordinate is zero. Loss contours (ellipses) most often first touch these corners. The L2 circle is smooth with no such corners, so it is touched on its side where both weights stay nonzero.
Why does the exact rearrangement of the L2 update pull out a factor?
Start from . Distribute the : . Now collect the two terms — — which leaves . The appears purely from factoring out of those two terms, and it shows the weight is decayed before the data gradient acts.
Why is L2 penalty preferred when you want smooth, well-conditioned weights rather than feature selection?
The squared term is differentiable everywhere and shrinks weights proportionally, spreading influence across correlated features instead of arbitrarily zeroing some. That yields stable, smooth solutions — useful when all features carry a little signal.
Why does dropout act like an ensemble even though you train just one network?
Every mini-batch samples a different random sub-network (a different subset of neurons active). Over training these share weights but explore many architectures, so the final network behaves like an average over exponentially many thin networks — the Ensemble Methods effect without training many models.
Why can L1 be seen as automatic feature selection but L2 cannot?
L1 pushes uninformative weights to exactly zero, so those features drop out of the model entirely — a selection decision. L2 keeps every feature with a small nonzero weight, so nothing is truly removed. This links L1 to Feature Engineering and interpretability.
Why does the constant-subtraction behaviour of L1 make small weights vanish but leave large weights mostly intact?
Each step removes a fixed (soft thresholding). A weight smaller than that step is pushed past zero and clamped to zero; a large weight loses the same fixed amount, which is a negligible fraction of its size, so it survives.
Why do we tune (and ) with a validation set rather than pick them from training loss?
Training loss always prefers less regularization (it can fit noise). Only held-out data via Cross-Validation reveals the value that best generalizes; picking on training error would drive and reintroduce overfitting.
Edge cases
What does the L2 update do when ?
It collapses to plain gradient descent — no decay, no penalty, full risk of overfitting. means "regularization off."
What happens to every weight if exactly (and the data gradient is zero)?
The decay factor , so is wiped to zero in a single step. This is a degenerate, too-aggressive setting; you normally keep .
What is the L1 subgradient at , and why do we specifically pick out of the allowed range ?
The classical derivative is undefined there, so any value in is a legal subgradient. We pick because at a weight already sitting at zero the data gradient alone decides its fate: a nonzero subgradient would keep nudging the weight off zero and destroy sparsity, whereas lets a genuinely useless weight rest at zero. That deliberate choice is what makes L1 a sparsity engine.
What does dropout with reduce to?
No neurons are ever dropped and the scale factor , so it is an ordinary network with no dropout at all — the regularization is switched off.
What does dropout with do, and why is it useless?
Almost every neuron is zeroed every pass, so no signal flows forward and the scale blows up the few survivors. Learning stalls — the network has essentially no active capacity.
If a feature column is exactly duplicated, how do L1 and L2 treat the pair differently?
L2 splits the weight roughly evenly between the two copies (it dislikes any single large weight). L1 tends to keep one copy and zero the other, since concentrating the weight on one costs the same L1 penalty but is a corner solution.
Recall Quick self-test
One-line L1-vs-L2 discriminator ::: L1 subtracts a constant → exact zeros → sparsity; L2 multiplies by → proportional shrink → smooth small weights. The one dropout rule people get wrong ::: With inverted dropout you scale kept activations by during training and do nothing at test time; classic dropout instead scales by at test time.