Ek single weightw lo aur sabse simple problem lo (orthonormal features taaki features decouple ho jayein). Cost hai:
J(w)=21(w−z)2+λ∣w∣
jahan z us feature ka OLS solution hai (woh value jo w bina penalty ke leta).
Yeh step kyun? Orthonormal features ke saath har coordinate independently optimize hota hai, toh ek w ko samajhna sab kuch bata deta hai.
Ab minimize karo. Absolute value 0 par differentiable nahi hai, toh hum do cases handle karte hain.
Case 3: −λ≤z≤λ. Upar wale dono cases consistent nahi hain, aur minimum kink par baith jaata hai w=0.
Isi setting mein Ridge se compare karo: 21(w−z)2+λw2 minimize karne par milta hai w=1+2λz — hamesha shrunk, lekin kabhi exactly zero nahi jab tak z=0 na ho.
Sλ(z)=sign(z)max(∣z∣−λ,0): λ se zero ki taraf shrink karo, aur exactly 0 par clamp karo agar magnitude λ se kam ho.
Recall L1 sparsity kyun deta hai lekin L2 nahi?
L1 ka constraint region (diamond) axes par corners rakhta hai; MSE contours wahan touch karte hain, coordinates ko 0 force karte hain. L2 ka circle smooth hai — koi corner nahi — toh weights shrink hote hain lekin nonzero rehte hain.
Recall Feynman: 12-saal ke bachche ko samjhao
Socho tum trip ke liye pack kar rahe ho lekin tumhara bag ek strict weight limit rakhta hai, aur har item jo tum add karo woh same "weight tax" leta hai chahe kitna bhi chota ho. Kyunki tax fixed hai per item, tum prefer karoge ki chote useless items bilkul baahar chod do thoda carry karne ka tax bharein. Lasso features ke saath yahi karta hai: jinhe "tax" ke laayak nahi samajhta unhe drop kar deta hai, sirf kuch important waale rakhta hai. Isliye model chota aur simple banta hai.
Lasso loss mein kaun sa penalty term add hota hai?
λ∑j∣wj∣=λ∥w∥1 (weights ka L1 norm).
Lasso exactly-zero weights kyun produce karta hai jabki Ridge nahi?
L1 ka constraint region ek diamond hai jisme axes par corners hain; loss contours pehle ek corner (zero) par touch karte hain. Ridge ka circle smooth hai, toh exact zeros nahi.
Orthonormal features ke liye soft-thresholding solution likho.
w⋆=sign(z)max(∣z∣−λ,0), jahan z OLS coefficient hai.
Lasso mein λ→0 aur λ→∞ par kya hota hai?
λ→0: OLS recover hota hai. λ→∞: saare penalized weights 0 ho jaate hain (model mean predict karta hai).
Lasso se pehle features standardize kyun karne chahiye?
L1 penalty saare coefficients ko equally treat karti hai; alag feature scales unequal effective penalization cause karti hai.
Plain gradient descent Lasso ke liye clean kyun nahi chal sakta?
∣w∣ 0 par non-differentiable hai; subgradients / coordinate descent / proximal (ISTA) methods use karo.
Elastic Net kya hai aur kyun use karein?
Ek mix λ1∥w∥1+λ2∥w∥22; yeh correlated features ke groups ko saath rakhta hai, Lasso ki instability fix karta hai.