Consider a single weightw with the simplest problem (orthonormal features so features decouple). The cost is:
J(w)=21(w−z)2+λ∣w∣
where z is the OLS solution for that feature (the value w would take with no penalty).
Why this step? With orthonormal features each coordinate optimizes independently, so studying one w tells us everything.
Now minimize. The absolute value is not differentiable at 0, so we handle two cases.
Case 1: w>0. Then ∣w∣=w, and
dwdJ=(w−z)+λ=0⇒w=z−λ
Valid only if w>0, i.e. z>λ.
Case 2: w<0. Then ∣w∣=−w, and
dwdJ=(w−z)−λ=0⇒w=z+λ
Valid only if z<−λ.
Case 3: −λ≤z≤λ. Neither of the above is consistent, and the minimum sits at the kink w=0.
Compare Ridge in the same setting: minimizing 21(w−z)2+λw2 gives w=1+2λz — always shrunk, but never exactly zero unless z=0.
Sλ(z)=sign(z)max(∣z∣−λ,0): shrink toward zero by λ, and clamp to exactly 0 if the magnitude is below λ.
Recall Why does L1 give sparsity but L2 does not?
L1's constraint region (diamond) has corners on the axes; MSE contours touch it there, forcing coordinates to 0. L2's circle is smooth — no corners — so weights shrink but stay nonzero.
Recall Feynman: explain to a 12-year-old
Imagine you're packing for a trip but your bag has a strict weight limit, and each item you add costs the same "weight tax" no matter how small it is. Because the tax is fixed per item, you'd rather leave small useless items completely out than pay tax to carry a bit of them. Lasso does this with features: it drops the ones not worth their "tax," keeping only a few important ones. That's why the model ends up short and simple.
Why does Lasso produce exactly-zero weights while Ridge does not?
L1's constraint region is a diamond with corners on the axes; the loss contours first touch at a corner (zero). Ridge's circle is smooth, so no exact zeros.
Write the soft-thresholding solution for orthonormal features.
w⋆=sign(z)max(∣z∣−λ,0), where z is the OLS coefficient.
What happens as λ→0 and as λ→∞ in Lasso?
λ→0: recovers OLS. λ→∞: all penalized weights become 0 (model predicts the mean).
Why must features be standardized before Lasso?
The L1 penalty treats all coefficients equally; different feature scales cause unequal effective penalization.
Why can't plain gradient descent be used cleanly for Lasso?
∣w∣ is non-differentiable at 0; use subgradients / coordinate descent / proximal (ISTA) methods instead.
What is Elastic Net and why use it?
A mix λ1∥w∥1+λ2∥w∥22; it keeps groups of correlated features together, fixing Lasso's instability.
Dekho, Lasso ka simple funda ye hai: normal regression sirf training error kam karta hai, isliye kabhi kabhi weights bahut bade ho jaate hain aur model noise ko bhi fit karne lagta hai — yani overfitting. Lasso ek extra "tax" lagata hai jo weights ki absolute values ka sum hai, λ∑∣wj∣. Jitna bada λ, utna zyada weights zero ki taraf khinchte hain.
Sabse important baat: Lasso kuch weights ko bilkul zero bana deta hai. Iska matlab wo feature model se hat gaya — automatic feature selection. Ridge (L2) sirf chhota karta hai, zero nahi karta. Iska reason geometry hai: L1 ka region ek diamond hota hai jiske corners axes par hote hain, aur loss ke contours usi corner pe touch karte hain jaha coordinate zero hota hai. L2 ka circle smooth hota hai, isliye zero nahi milta.
Formula wise, ek simple case mein solution nikalta hai soft-thresholding: w=sign(z)max(∣z∣−λ,0). Matlab har coefficient ko λ jitna zero ki taraf khisko, aur agar wo zero cross karne lage to seedha zero pe clamp kar do. Ye hi wo trick hai jo sparsity deti hai.
Do practical baatein yaad rakhna: (1) Lasso lagane se pehle features ko standardize karo warna scale ki wajah se penalty unfair ho jayegi. (2) λ ko cross-validation se choose karo — bahut bada λ underfit kar dega. Agar features correlated hain to Lasso thoda unstable hota hai, tab Elastic Net use karo. Bas itna samajh gaye to Lasso clear hai!