Foundations — Implementing models from scratch

Everything on the parent page Implementing models from scratch is built out of a small pile of symbols. This page opens that pile and hands you each piece, in the order you need them. Nothing here assumes you have seen the notation before.

1. A number, a vector, a matrix

Before any layer, we must agree on what the letters even hold.

We write a vector in bold lowercase ($\mathbf{x}$) and a matrix in bold uppercase ($\mathbf{W}$). A plain italic letter with a subscript, like $x_j$, means "the $j$-th number inside the vector $\mathbf{x}$." So $\mathbf{x}$ is the whole stack; $x_2$ is the second box.

Why the topic needs this: the input to a network (say, the 784 pixel-brightnesses of an image) is a vector. A layer's parameters are a matrix. Reading $\mathbf{Wx}$ correctly is impossible until "vector" and "matrix" mean something concrete.

For a matrix we write $\mathbf{W}\in {\mathbb{R}}^{m\times n}$: a grid with $m$ rows and $n$ columns. Read it left-to-right as "rows by columns," always.

2. Multiplying a matrix by a vector

This is the single most-used operation on the parent page. Let us build it from dots.

A matrix times a vector just does this dot product once per row. Row $i$ of $\mathbf{W}$ dotted with $\mathbf{x}$ produces entry $i$ of the answer.

Why the topic needs this: $\mathbf{z}=\mathbf{Wx}+\mathbf{b}$ is the forward pass of one layer. If dot products are foggy, every equation downstream is fog. See 2.3.01-Matrix-Operations for the mechanical drill.

3. The transpose ${\mathbf{W}}^T$

Why the topic needs this: in backprop the parent writes $\frac{\partial L}{\partial \mathbf{X}}={\mathbf{W}}^T\delta$. The forward pass sent information $\mathbf{X}\to \mathbf{Z}$ using $\mathbf{W}$; to push the gradient back the other way, $\mathbf{Z}\to \mathbf{X}$, we reuse the same numbers but with rows and columns swapped. The transpose is the "reverse gear" of a matrix multiply. It also fixes shapes: $\mathbf{W}$ is $m\times n$, so ${\mathbf{W}}^T$ is $n\times m$, which is exactly the shape that turns an $m$-tall gradient back into an $n$-tall one.

4. Adding a bias, and broadcasting

When we process many samples at once, the input becomes a matrix $\mathbf{X}$ (one column per sample). We still want to add the same $\mathbf{b}$ to every column. Copying $\mathbf{b}$ automatically across all columns is called broadcasting.

5. The non-linearity $\sigma$

The little tick mark in ${\sigma }^{\prime }$ means the derivative of $\sigma$ — how steeply the curve rises at a point. We meet derivatives next.

6. The derivative and the gradient

Neural networks have millions of inputs (the parameters), not one. The multi-input version of a derivative is the gradient.

Here $L$ is the loss — a single number measuring how wrong the network's prediction is (bigger = worse). And $\mathbf{\theta }$ (Greek "theta", bold because it's all the parameters bundled together) stands for every $\mathbf{W}$ and $\mathbf{b}$ in the network at once.

Why the topic needs this: learning is rolling downhill on the loss surface. The whole "backward pass" exists only to compute this gradient. Foundations in 4.2.02-Gradient-Descent.

7. Partial derivatives and the chain rule

The symbol $\delta$ (Greek "delta") is just a nickname for the gradient that has already flowed back to a layer's pre-activation, $\delta =\frac{\partial L}{\partial \mathbf{Z}}$. And $\odot$ means element-wise multiply — multiply two equal-shaped grids box-by-box (no dot-product summing), used when a curve's local slope ${\sigma }^{\prime }$ gates the incoming gradient.

8. Averaging, batches, and the learning rate

The prerequisite map

Equipment checklist

Test yourself — reveal only after you have an answer.