Exercises — L1 (Lasso) regularization

• (a) is the L1 / Lasso penalty — the sum of absolute values. Its constraint region is a diamond with corners on the axes, so it produces exact zeros.
• (b) is the L2 / Ridge penalty — sum of squares. Its region is a smooth circle: weights shrink but never hit exactly zero (unless already zero).
• (c) is Elastic Net: the L1 part still supplies the corner, so it can produce zeros, while the L2 part keeps correlated features grouped.

Here $z=4>0$ and $|z|=4>\lambda =1.5$, so we are in the positive case: shrink by $\lambda$. $S_{1.5}(4)=\mathrm{sign}(4)\max (4-1.5,0)=+1\cdot 2.5=\boxed{2.5}$

Why coordinate-by-coordinate works here. "Orthonormal features" means the feature columns are perpendicular unit vectors, so the data-fit term splits into a clean sum with no cross-terms: $\frac{1}{2}\Vert Xw-y{\Vert }^2=\frac{1}{2}{\sum }_j(w_j-z_j{)}^2+\text{const}$, where $z_j$ is the OLS coefficient for feature $j$. Add the L1 penalty $\lambda {\sum }_j|w_j|$ and the whole cost becomes a sum over $j$ — each term $\frac{1}{2}(w_j-z_j{)}^2+\lambda |w_j|$ depends only on its own $w_j$. Minimizing a sum of independent one-variable problems is the same as minimizing each separately, and each is exactly the soft-threshold. That is why coordinates decouple.

Apply $S_{0.9}(z_j)=\mathrm{sign}(z_j)\max (|z_j|-0.9,0)$ per coordinate:

• $w_1=\mathrm{sign}(2.0)\max (2.0-0.9,0)=+1.1$
• $w_2=\mathrm{sign}(-0.3)\max (0.3-0.9,0)=0$ (dropped: $0.3<0.9$)
• $w_3=\mathrm{sign}(0.9)\max (0.9-0.9,0)=0$ (dropped: $0.9\ngtr 0.9$)
• $w_4=\mathrm{sign}(-1.5)\max (1.5-0.9,0)=-0.6$

Result $w=[1.1,0,0,-0.6]$. Features 1 and 4 survive; 2 and 3 are eliminated. Note the boundary case $z_3=0.9=\lambda$: since we need $|z|>\lambda$ strictly, it clamps to zero.

Lasso: $S_{0.4}(1.6)=\max (1.6-0.4,0)=1.2$. Ridge derivative, step by step. We minimize $f(w)=\frac{1}{2}(w-z{)}^2+\lambda w^2$ by setting $f^{\prime }(w)=0$.

• The $\frac{1}{2}$ is placed there on purpose: differentiating $\frac{1}{2}(w-z{)}^2$ uses the power rule (bring the exponent $2$ down) and chain rule (times the inside derivative $1$), giving $\frac{1}{2}\cdot 2\cdot (w-z)\cdot 1=(w-z)$. The $\frac{1}{2}$ and the $2$ cancel — that is why we write the $\frac{1}{2}$, to get a clean derivative.
• Differentiating $\lambda w^2$ likewise brings the exponent $2$ down: $\lambda \cdot 2w=2\lambda w$.
• So $f^{\prime }(w)=(w-z)+2\lambda w=0\Rightarrow w(1+2\lambda )=z\Rightarrow w=\frac{z}{1+2\lambda }=\frac{1.6}{1.8}\approx 0.8\overline{8}$.

Lasso subtracts a constant ($0.4$); Ridge multiplies by a constant factor ($1/1.8$). Neither is zero here, but only Lasso could reach zero (if $|z|\le \lambda$).

$w^{\star }(\lambda )=\max (1.0-\lambda ,0)$ (since $z>0$).

• For $0\le \lambda <1.0$: $w^{\star }=1.0-\lambda$, a straight line decreasing with slope $-1$.
• At $\lambda =1.0$: $w^{\star }=0$.
• For $\lambda >1.0$: stays at $0$.

The feature dies exactly at $\lambda =|z|=1.0$.

Reading the figure below (s01): the horizontal axis is the penalty strength $\lambda$ (from $0$ to $2$); the vertical axis is the resulting Lasso weight $w^{\star }$. The magenta curve starts at $w^{\star }=1$ when $\lambda =0$ (no penalty ⇒ full OLS value), slides down as a straight ramp of slope $-1$ (violet arrow), hits zero at the orange dot where $\lambda =1$, and then stays flat on the axis. This ramp-then-flat shape is the "regularization path" for one feature.

Minimizing MSE subject to a budget $\Vert w{\Vert }_1\le t$ means: expand the elliptical MSE contours (centered at the OLS solution) outward until they first touch the constraint region.

• The L1 diamond has sharp corners sitting on the axes. A corner sticks out, so an incoming ellipse overwhelmingly touches it there first — and a corner has one coordinate $=0$. That coordinate is dropped.
• The L2 circle is smooth: no corner sticks out, so the first touch is at a generic point where both coordinates are nonzero. Zero-touching is a measure-zero coincidence.

Reading the figure below (s02): both axes are the two weights, $w_x$ (horizontal) and $w_y$ (vertical). The magenta diamond $|w_x|+|w_y|\le t$ has four corners exactly on the axes; the violet dashed circle is the L2 budget of the same size. The orange dot up-and-right is the OLS solution, and the faint orange rings around it are MSE level sets (contours of equal error — each ring is a set of weight pairs giving the same training error, growing outward). Watch which shape the smallest ring reaches first: it lands on the magenta diamond corner (marked, where $w_x=0$), not on a smooth side. That single touch is why a coordinate becomes exactly zero under L1.

(a) Two correlated features carry almost the same information. The L1 penalty pays a "tax" per unit of coefficient magnitude regardless of feature, so putting all the weight on one feature costs the same as splitting it — but concentrating on one lets the other drop to zero (a corner), which the diamond geometry prefers. Which of the twins wins is decided by tiny sampling noise, so it flips between splits ⇒ instability. (b) Elastic Net: add an L2 term, minimizing ${\lambda }_1\Vert w{\Vert }_1+{\lambda }_2\Vert w{\Vert }_2^2$. The L2 part is smooth and prefers to spread weight across correlated features (it penalizes large individual weights quadratically), so the twins get similar nonzero weights and are kept together and stably. This is the grouping effect.

Scaling a feature by $1000\times$ (meters → millimeters) divides its coefficient by $1000$: $z_a^{\prime }=z_a/1000=0.0006$.

• Meters: $S_{0.01}(0.6)=\max (0.6-0.01,0)=0.59\ne 0$ → survives.
• Millimeters: $S_{0.01}(0.0006)=\max (0.0006-0.01,0)=0$ → dropped.

The same feature is kept or dropped depending only on its units! The L1 penalty $\lambda |w_j|$ treats all coefficients equally, so a differently-scaled coefficient is penalized differently. Standardize (zero mean, unit variance) so every feature faces the same effective tax.

This is the Coordinate Descent update: it isolates one coordinate, treats the rest as fixed, and solves a one-dimensional Lasso problem — which is exactly soft-thresholding, then rescaled by $a_j$. $S_{1.0}(5.0)=\max (5.0-1.0,0)=4.0$ $w_j\leftarrow \frac{4.0}{4.0}=\boxed{1.0}$ WHY the $\frac{1}{a_j}$? ${\rho }_j$ is a raw correlation; dividing by $a_j={\sum }_i{x}_{ij}^2$ normalizes it into a coefficient scale. The soft-threshold is applied on the correlation scale (numerator), which is why the non-orthonormal case still reduces to $S_{\lambda }$.

Proximal Gradient (ISTA/FISTA) splits the objective into smooth (logistic loss, differentiable) + non-smooth (L1). It gradient-steps the smooth part, then applies the L1 proximal operator, which is soft-thresholding with threshold $\eta \lambda$. Threshold $=\eta \lambda =0.5\times 1.0=0.5$. $S_{0.5}(-0.7)=\mathrm{sign}(-0.7)\max (0.7-0.5,0)=-1\cdot 0.2=\boxed{-0.2}$ The sign is preserved, the magnitude shrunk by the threshold — same mechanism as linear Lasso, because the L1 term is independent of which loss (MSE or logistic) it is attached to.

A feature $j$ dies when $\lambda =|z_j|$ (the shrink amount finally cancels its magnitude):

• Feature 2 ($|z_2|=0.4$) dies first at $\lambda =0.4$.
• Feature 3 ($|z_3|=1.5$) dies at $\lambda =1.5$.
• Feature 1 ($|z_1|=2.0$) dies last at $\lambda =2.0$.

Death order = increasing $|z_j|$: the smallest-signal feature drops first. This is the "regularization path," and where to stop along it is chosen by Cross-Validation — trading bias against variance.

$\lambda$ does an orthonormal feature with OLS coefficient $z$ die? Exactly at $\lambda =|z|$ — the point where the shrink amount equals the coefficient's magnitude.

Each coordinate's 1-D subproblem is a soft-threshold on the correlation ${\rho }_j$, rescaled by $1/a_j$: $w_j=\frac{1}{a_j}{S}_{\lambda }({\rho }_j)$.

$\mathrm{sign}(0)$ and what does $S_{\lambda }(0)$ give? $\mathrm{sign}(0)=0$, and $S_{\lambda }(0)=0$: a coefficient already at zero stays at zero.