Foundations — Regularization — L1 (lasso), L2 (ridge), dropout

Before you can read the parent note on L1, L2, and dropout, you must be able to read every squiggle it uses without pausing. This page builds each one from absolute zero, in the order they depend on each other.

1. A weight $w$ — the knob

Picture a mixing desk. Each slider is a $w_i$. Turning slider $i$ up means "pay more attention to input number $i$." The whole collection of sliders is written as a bold $w$ (a vector — an ordered list of numbers).

The letter $b$ that appears in the parent's model ($C_D=w_1\alpha +\cdots +b$) is the bias — one extra slider that shifts everything up or down by a constant, independent of any input. It is usually not penalized.

2. Prediction $\hat{y}$ vs. truth $y$ — the hat means "guess"

Picture two dots on a number line: a black dot at the true value $y$, and a hollow dot at the guess $\hat{y}$. The gap between them is the model's mistake on this one example.

3. Summation $\sum$ — "add up a list"

Picture a shopping receipt: each line is one $a_n$, and $\sum$ is the total at the bottom.

The topic needs $\sum$ because a penalty must combine all the weights into a single number to add to the score — you can't penalize a whole desk of sliders without summing over them.

4. The loss $\mathcal{L}$ — the badness score

Picture a valley. The horizontal position is your setting of the weights; the height of the ground is the loss. Training rolls a ball downhill toward the lowest point.

The parent note splits the loss into two named pieces:

The whole point of regularization is: don't minimize ${\mathcal{L}}_{\text{data}}$ alone; minimize ${\mathcal{L}}_{\text{total}}$, which also cares about knob size.

The tiny $\ell (y_n,{\hat{y}}_n)$ inside the sum is the per-example loss — the badness on just one flight $n$. Summing all of them and dividing by $N$ (the number of examples) gives the average, which is ${\mathcal{L}}_{\text{data}}$.

5. Magnitude bars: $|w|$ and $\Vert w\Vert$ — "how big, ignore direction"

Picture the point $w=(w_1,w_2)$ on a map. $\Vert w{\Vert }_2$ is the crow-flies distance to the origin. $\Vert w{\Vert }_1$ is the taxi-cab distance (you must travel along the grid streets).

6. The slope $\frac{\partial \mathcal{L}}{\partial w_i}$ — which way is downhill

We need a derivative (and not, say, just evaluating the loss) because to roll downhill you must know the direction of down at your current spot — that direction is exactly the slope.

The curly $\partial$ (instead of a straight $d$) is a flag: "there are several knobs; I'm changing just this one and holding the rest still."

The stacked-together partials for all weights form the gradient, written $\nabla \mathcal{L}$ (the upside-down triangle "nabla"). It is just the full list of slopes — the complete "downhill arrow."

7. The gradient-descent step: $\eta$, $\lambda$, and the arrow $\leftarrow$

The superscript in $w^{(t)}$ just means "the weight at step number $t$" — $t=0$ is the start, $t=1$ after one update, and so on. It is a time stamp, not a power.

8. Probability $p$ and expectation $\mathbb{E}$ — the language of dropout

The "$m_i\sim \text{Bernoulli}(1-p)$" in the parent means: $m_i$ is a random on/off switch that is $1$ with probability $1-p$ and $0$ otherwise. Multiplying $h_i$ by $m_i$ keeps or kills that neuron.

Prerequisite map

This feeds directly into the parent topic. Related roads: Gradient Descent Variants refines the update rule, Feature Engineering and Overfitting Detection motivate why we penalize, Cross-Validation picks $\lambda$, and Ensemble Methods plus Bayesian Inference give deeper reasons dropout works.

Equipment checklist