Intuition The ONE core ideaLasso is ordinary line-fitting with a small "tax" charged on how big the weights get, where the tax is the plain sum of the sizes of the weights . Because that particular tax has a sharp corner exactly at zero, it doesn't just shrink weights — it snaps the useless ones all the way down to exactly zero , deleting features automatically.
Before you can read the parent note, you must be able to hear every symbol in it without pausing. This page walks through each one, from the most basic to the sharpest, always attaching a picture. If the parent said "the L1 norm is a diamond," this page is where the diamond is actually drawn.
Definition What a single row of data isx i list of measurements about one example (one house: its size, its age, its number of rooms). y i single number we want to predict about that example (its price). The little subscript i i
Picture a spreadsheet. Each row is one example. The columns before the last are the x y
Intuition Why we need thisEverything in regression is about learning a rule that turns the row x i y i ( x i , y i )
n how many examples (rows) we have.p how many features (measurement columns) each example has.
Definition Weights = importance dialsA weight w j j w j zero means "this feature is ignored completely."
We stack all the weights into a list w = ( w 1 , w 2 , … , w p )
w 0 w 0 bias — a baseline offset added to every prediction, independent of any feature. It's the "starting height" of the line before any feature pushes it. The complete prediction for example i y ^ i = w 0 + w 1 x i 1 + w 2 x i 2 + ⋯ + w p x i p = w 0 + w ⊤ x i w ⊤ x i weighted sum ) collects only the feature part:
w ⊤ x i = w 1 x i 1 + w 2 x i 2 + ⋯ + w p x i p ⊤
Look at the figure: each blue feature value gets scaled by its yellow dial, the scaled pieces drop into one green sum, and the baseline w 0 is the prediction y ^ i = w 0 + w ⊤ x i
Intuition Why a weighted sum (plus a baseline)?The weighted sum is the simplest rule that lets every feature have its own adjustable influence, and the bias w 0 finding the right dial settings (and baseline) . The whole topic of Lasso is a rule for choosing those dials so that many of them end up at zero — and, importantly, Lasso taxes only the feature dials w 1 , … , w p never the baseline w 0
Definition Error of one exampleThe error on example i y i − y ^ i = y i − ( w 0 + w ⊤ x i )
Definition Why we square itWe use ( y i − y ^ i ) 2 squared — for two reasons: squaring makes every error positive (a miss is a miss whether above or below), and it punishes big misses much harder than small ones (a miss of 4 costs 16, a miss of 2 costs only 4).
The figure shows dots (real data) and a line (our guesses). Each red vertical stick is one error; we square each stick's length and add them up. A better line makes the total shorter.
Definition Mean Squared Error (MSE) and the "half" versionThe true Mean Squared Error is the plain average of the squared errors:
MSE = n 1 ∑ i = 1 n ( y i − ( w 0 + w ⊤ x i ) ) 2 ∑ i = 1 n add up over all n ; dividing by n average (so more data doesn't inflate the number).
Intuition Why the topic needs this termThis is the "data fit" half of Lasso's cost. Minimizing it alone is Ordinary Least Squares (OLS) — the plain line of best fit. Lasso = data-fit plus a tax . You cannot understand the tax's effect without first knowing what it's being added to.
Definition Absolute value = size, sign thrown away∣ w j ∣ distance of w j , always positive. ∣3∣ = 3 ∣ − 3∣ = 3
∥ w ∥ 1 all the weights:
∥ w ∥ 1 = ∣ w 1 ∣ + ∣ w 2 ∣ + ⋯ + ∣ w p ∣ = ∑ j = 1 p ∣ w j ∣ ∥ ⋅ ∥ 1 which recipe — sum-of-sizes. (The sum starts at j = 1 w 0
Intuition Why sizes and not the raw numbers?If we added the raw weights, a big positive dial and a big negative dial would cancel, and a wild model would look "small." Taking sizes first means every large dial always increases the tax , no cancelling. That is exactly the honesty we want when penalizing complexity.
Definition The other recipe: L2 norm (squared)Ridge uses ∥ w ∥ 2 2 = w 1 2 + w 2 2 + ⋯ + w p 2 squares instead of sizes. Same goal (measure bigness), different shape. The whole "diamond vs circle" story is about these two recipes drawing different shapes.
The figure draws, for two weights ( w 1 , w 2 ) L1 recipe that set is a diamond with sharp corners sitting on the axes; for the L2 recipe it's a smooth circle . Notice: the diamond's corners are exactly the points where one weight is zero. That corner is the entire reason Lasso produces zeros — hold onto this picture, the parent note leans on it hard.
Definition The regularization strengthλ λ how heavy is the tax on weight-size? λ = 0 λ
J ( w ) J ( w ) objective (also called the cost or loss ) — one single number that scores how bad a choice of weights w J searching for the w J ( w ) . The letter J ( w )
Intuition Why one knob controls everythingλ balance between the two terms above — "fit the data" (half-MSE) versus "stay simple" (small weights). Turning it up trades a little training accuracy for a lot of simplicity. The parent note's "sweet spot," chosen by Cross-Validation , is just the best setting of this one knob.
z single dial, imagine we've frozen all the other weights and ask: "what should this one weight be?" With no tax, the data-fit term alone is minimized at some specific number — call it z z
Concretely, in the simplest decoupled setting the data-fit term for one weight w 2 1 ( w − z ) 2 z ( w − z ) w = z z
sign ( z ) + 1 z − 1 0 remembers the direction of a number while we mess with its size.
max ( a , 0 ) a a 0 floor at zero — it refuses to go negative.
Intuition Why the topic needs all threeThe soft-threshold, which turns the un-penalized target z w ⋆ w ⋆ = S λ ( z ) = sign ( z ) max ( ∣ z ∣ − λ , 0 ) keep the original direction of z λ Every symbol now has a picture: z sign ∣ z ∣ λ max ( ⋅ , 0 )
A kink is a sharp corner in a graph where it suddenly changes direction. The graph of ∣ w ∣ V : it comes down with slope − 1 0 + 1
Look at the figure: the yellow V is ∣ w ∣ w = 0 w 2 − 1 + 1 no single slope — that abrupt change is the kink. The parabola, by contrast, glides through zero with a smooth, gentle bottom.
Intuition Why this matters more than anythingSmooth curves have one clear slope at every point, which lets plain gradient descent work. But at the bottom of the V there is no single slope — it's − 1 + 1 trap that catches weights and holds them at zero. Smooth penalties (Ridge's parabola w 2 Coordinate Descent and Proximal Gradient (ISTA/FISTA) instead of plain gradients.
prediction w0 plus w-dot-x
error = truth minus prediction
squared error then averaged = half MSE
L1 penalty = lambda times L1 norm
cost J = half MSE plus L1 penalty
kink at zero in the V shape
soft-threshold uses sign and max
some weights become exactly zero = Feature Selection
The map reads top-to-bottom, and it is exactly the equation from Section 5 drawn as a graph:
J ( w ) = 2 n 1 ∑ i = 1 n ( y i − ( w 0 + w ⊤ x i ) ) 2 + λ ∑ j = 1 p ∣ w j ∣ λ J ( w )
Once these symbols are second nature, the parent L1 (Lasso) regularization note (or its English twin) shows the full derivation of the soft-threshold, the diamond geometry, and worked examples. Related paths: Ridge (L2) Regularization uses the circle instead of the diamond, Elastic Net blends both, and the same penalty attaches to Logistic Regression . Choosing λ Bias-Variance Tradeoff settled by Cross-Validation .
What does the subscript i x i "Example number i
What does w 0 + w ⊤ x i A baseline plus a weighted sum: each feature times its dial, all added up onto the bias — the model's prediction.
What is the bias w 0 A baseline offset added to every prediction; it appears as the w 0 y ^ i = w 0 + w ⊤ x i
Why do we square the error instead of just adding it? To make every miss positive and to punish big misses far more than small ones.
What does ∑ i = 1 n Add the following expression up over all n
What is the difference between the MSE and the term used in the cost? MSE = n 1 ∑ ( ⋅ ) 2 2 n 1 ∑ ( ⋅ ) 2 2 1
What does J ( w ) The objective / cost / loss — one number scoring how bad the weights w
What is ∥ w ∥ 1 The L1 norm — the sum of the absolute values (sizes) of all the feature weights.
Why take absolute values before summing the weights? So big positive and negative dials can't cancel; every large weight always raises the tax.
What shape is the L1 budget region for two weights, and why does it matter? A diamond with corners on the axes; contours touch those corners, forcing weights to zero.
Write out the full Lasso cost J ( w ) J ( w ) = 2 n 1 ∑ i = 1 n ( y i − ( w 0 + w ⊤ x i ) ) 2 + λ ∑ j = 1 p ∣ w j ∣
What does λ The tax rate — the balance between fitting the data and keeping weights small.
Where does z It is the un-penalized (OLS) value a single weight would take with no tax — the bottom of that coordinate's data-fit bowl 2 1 ( w − z ) 2
Read sign ( z ) max ( ∣ z ∣ − λ , 0 ) Keep the sign/direction, subtract the tax from the size, and clamp to zero if it would go negative.
Why does ∣ w ∣ Its V-shape has a kink — no single slope at zero — so plain gradient descent can't cleanly land there.
