Exercises — Elastic Net regularization

• First term = data-fit / mean-squared error (the $\frac{1}{2n}\sum (\dots {)}^2$). It wants the predictions to match $y$.
• $\alpha \Vert \mathbf{w}{\Vert }_1$ = the L1 penalty; the sum of absolute weights. Its corners on the axes create sparsity (exact zeros).
• $\frac{1-\alpha }{2}\Vert \mathbf{w}{\Vert }_2^2$ = the L2 penalty; the sum of squared weights. It shrinks and groups correlated features.
• $\lambda \ge 0$ = how much total regularization (the overall strength knob).
• $\alpha \in [0,1]$ = what flavour (the mix). $\alpha =1\Rightarrow$ Lasso, $\alpha =0\Rightarrow$ Ridge.

Compare $|z|$ to $\gamma =0.5$.

• $S(-0.2,0.5)$: $|-0.2|=0.2<0.5$ → inside the dead zone → $\boxed{0}$.
• $S(1.3,0.5)$: $|1.3|=1.3>0.5$ → survives; pull toward zero by $0.5$: $\mathrm{sign}(1.3)\cdot (1.3-0.5)=+0.8$ → $\boxed{0.8}$.

With $\alpha =0$ the threshold is $\lambda \alpha =0$, so $S({\rho }_j,0)={\rho }_j$ (no dead zone at all). The update becomes $w_j={\rho }_j/(1+\lambda )$ — pure proportional shrink. It reaches $0$ only if ${\rho }_j$ is exactly $0$. So generically no zeros — that is exactly Ridge behaviour.

Threshold $=\lambda \alpha =2\cdot 0.5=1.0$. Since $|0.7|=0.7<1.0$, the numerator $S(0.7,1.0)=0$. Therefore $w_j=0/(1+2\cdot 0.5)=0/2=\boxed{0}$. The feature is dropped — its residual correlation cannot beat the L1 threshold.

• Threshold $=\lambda \alpha =0.25$. $|-1.2|=1.2>0.25$ → survives.
• Numerator $S(-1.2,0.25)=\mathrm{sign}(-1.2)\cdot (1.2-0.25)=-0.95$.
• Denominator $=1+\lambda (1-\alpha )=1+1\cdot 0.75=1.75$.
• $w_j=-0.95/1.75=\boxed{-0.542857\dots }$ (about $-0.543$). Note the sign of ${\rho }_j$ carries straight through.

Along the flat data direction the only tie-breaker is the L2 term $w_1^2+w_2^2$ subject to $w_1+w_2=1$. Minimising a sum of squares with a fixed sum happens at equality: $w_1=w_2$. So $w_1=w_2=\boxed{0.5}$. This is the grouping effect; see Feature Correlation & Multicollinearity. Check the minimisation: $w_1^2+(1-w_1{)}^2$ has derivative $2w_1-2(1-w_1)=0\Rightarrow w_1=0.5$.

• $\alpha =1$: threshold $=\lambda \cdot 1=\lambda$; denominator $=1+\lambda (1-1)=1$. So $w_j=S({\rho }_j,\lambda )$ — exactly the Lasso soft-threshold, no divisor.
• $\alpha =0$: threshold $=\lambda \cdot 0=0$; $S({\rho }_j,0)={\rho }_j$. Denominator $=1+\lambda$. So $w_j={\rho }_j/(1+\lambda )$ — pure Ridge proportional shrink. The single formula therefore contains both endpoints; $\alpha$ slides continuously between them.

With $\alpha =0.5$: threshold $=0.5\lambda$ grows, and divisor $=1+0.5\lambda$ grows.

• Effect 1 (numerator): the dead zone $[-0.5\lambda ,0.5\lambda ]$ widens, subtracting more from ${\rho }_j$; eventually $0.5\lambda >|{\rho }_j|$ and the weight snaps to $0$.
• Effect 2 (denominator): even before that, dividing by the growing $1+0.5\lambda$ shrinks the surviving value. Prediction: the weight shrinks continuously and then hits exactly $0$ once $\lambda \ge 2|{\rho }_j|$ (from $0.5\lambda =|{\rho }_j|$). This is the bias increasing / variance decreasing trade of Bias-Variance Tradeoff.

Zero happens when threshold $=|{\rho }_j|$: $\lambda \alpha =1\Rightarrow 0.5\lambda =1\Rightarrow \boxed{\lambda =2}$. At $\lambda =1.5$: numerator $S(1,0.75)=1-0.75=0.25$; denominator $1+1.5\cdot 0.5=1.75$; $w_j=0.25/1.75=\boxed{0.142857\dots }$. So just before dropping it is a small positive number, then it collapses to $0$ at $\lambda =2$ — continuous, not a jump.

Differentiate the smooth parts. The quadratic term: $\frac{d}{d{w}_j}[\frac{1}{2n}{\sum }_i(r_i-w_j{x}_{ij}{)}^2]=-\frac{1}{n}{\sum }_i{x}_{ij}(r_i-w_j{x}_{ij})=-{\rho }_j+w_j$ (using $\frac{1}{n}\sum x_{ij}^2=1$). The L2 term derivative: $\lambda (1-\alpha )w_j$. Handle the non-smooth $|w_j|$ via subgradient $\partial |w_j|=\mathrm{sign}(w_j)$ for $w_j\ne 0$, and $[-1,1]$ at $0$. Setting the subgradient to contain $0$: $0\in -{\rho }_j+w_j+\lambda (1-\alpha )w_j+\lambda \alpha \partial |w_j|.$ Case $w_j>0$: $\partial |w_j|=1$: $w_j(1+\lambda (1-\alpha ))={\rho }_j-\lambda \alpha$. Valid only if RHS $>0$, i.e. ${\rho }_j>\lambda \alpha$. Case $w_j<0$: $\partial |w_j|=-1$: $w_j(1+\lambda (1-\alpha ))={\rho }_j+\lambda \alpha$, valid if ${\rho }_j<-\lambda \alpha$. Case $w_j=0$: need $0\in -{\rho }_j+\lambda \alpha [-1,1]$, i.e. $|{\rho }_j|\le \lambda \alpha$. Stitching all three cases: $w_j=\frac{\mathrm{sign}({\rho }_j)\max (|{\rho }_j|-\lambda \alpha ,0)}{1+\lambda (1-\alpha )}=\frac{S({\rho }_j,\lambda \alpha )}{1+\lambda (1-\alpha )}.■$ Uses Coordinate Descent (optimise one $w_j$, hold the rest fixed).

• $x_3$ (noise): ${\rho }_3\approx 0$, so $|{\rho }_3|<\lambda \alpha$ → soft-threshold kills it → $w_3=0$. Both Lasso and Elastic Net drop it (L1 dead zone).
• $x_1,x_2$ (correlated, useful): Lasso's flat objective picks one arbitrarily and zeros the other → unstable across resamples. Elastic Net's denominator (L2) is strictly convex → splits the weight $\approx$ evenly → stable grouping. So: numerator/L1 removes junk; denominator/L2 stabilises the correlated group. That is precisely the "why not both" pitch.

Equality proof. Since $x_1=x_2$, swapping $w_1\leftrightarrow w_2$ leaves the data-fit and the L1 term unchanged, and leaves $w_1^2+w_2^2$ unchanged. So $J$ is symmetric in $(w_1,w_2)$. $J$ is strictly convex in $(w_1,w_2)$ because the L2 term $\frac{\lambda (1-\alpha )}{2}(w_1^2+w_2^2)$ is strictly convex for $\alpha <1$ (the data-fit and L1 are convex, adding a strictly convex term makes the whole strictly convex). A strictly convex function has a unique minimiser; a symmetric function's unique minimiser must lie on the symmetry axis $w_1=w_2$. Hence $w_1=w_2$. $■$ Shared value. Write $w_1=w_2=w$. Their combined feature is $x_1+x_2=2x_1$; treat the pair effectively so each carries residual correlation $\rho =0.8$ and each has its own L2 shrink. Solving the symmetric first-order condition gives each $w=\frac{S(\rho ,\lambda \alpha )}{1+\lambda (1-\alpha )}$ scaled by the split. Numerically: $S(0.8,0.5)=0.3$, divisor $=1.5$, single-coordinate value $0.2$. Since the pair shares the load, each gets $0.1$ (half of $0.2$). $\boxed{w_1=w_2=0.1}$, total $0.2$.

Penalty $P(\mathbf{w})=\lambda \alpha \Vert \mathbf{w}{\Vert }_1+\frac{\lambda (1-\alpha )}{2}\Vert \mathbf{w}{\Vert }_2^2$.

• $\Vert \mathbf{w}{\Vert }_1$ is convex but not strictly (it is linear along any orthant direction — flat pieces).
• $\Vert \mathbf{w}{\Vert }_2^2$ is strictly convex (its Hessian $2I$ is positive definite).
• For $\alpha <1$ the coefficient $\frac{\lambda (1-\alpha )}{2}>0$, so $P$ inherits strict convexity → unique minimiser even when the data-fit is flat.
• For $\alpha =1$ the strictly convex piece vanishes; $P$ is only convex, so along a data-flat direction the total objective is flat → infinitely many minimisers = Lasso instability on correlated features.

Recognition ::: L1 = dead-zone/threshold (numerator), L2 = divisor (denominator). Application ::: plug ${\rho }_j,\lambda ,\alpha$ into $w_j=S({\rho }_j,\lambda \alpha )/(1+\lambda (1-\alpha ))$, numerator first. Analysis ::: $\lambda$ = amount, $\alpha$ = flavour; weight drops to $0$ when $\lambda \alpha \ge |{\rho }_j|$. Synthesis ::: derive via subgradient; the kink at $0$ makes sparsity. Mastery ::: strict convexity ($\alpha <1$) = unique solution = grouping/stability.