Foundations — Building models with nn.Module

Before you can read the parent note Building models with nn.Module, you must own every symbol it throws at you. Below, each symbol is built from nothing: plain words → the picture → why the topic needs it. Read top to bottom; each item leans on the one above it.

1. A number, a vector, a matrix — the raw material

Everything a network touches is a grid of numbers. We need three names for three shapes of grid.

Why the topic needs this: The parent note writes torch.randn(32, 784) — that is a matrix with 32 rows and 784 columns. A network's job is to turn one grid of numbers into another grid of numbers, so we must be fluent in reading grid shapes.

2. The @ symbol — matrix multiplication

The parent's naive layer is x @ w + b. The @ is not "email at" — it means matrix multiply.

Suppose one input row is $x=[x_1,x_2]$ and the weight grid is $W=\left[\begin{array}{cc}{w}_{11}& w_{12}\\ w_{21}& w_{22}\end{array}\right].$ Then $x@W=[x_1{w}_{11}+x_2{w}_{21},x_1{w}_{12}+x_2{w}_{22}].$

Why the topic needs this: A nn.Linear layer is the operation $x@W+b$. Everything else — ReLU, residual blocks — is decoration around this core.

3. The + b bias — a shiftable floor

Why the topic needs this: Without a bias, every layer is forced to pass through zero when the input is zero. Bias lets the network say "even when nothing comes in, output should be $0.3$." In the parent, nn.Linear(in, out) secretly carries a bias of size out — that is where the "+128" and "+1" in the parameter count come from.

4. The activation — bending a straight line

If we only ever did $x@W+b$, stacking layers would collapse into one big multiply (a straight line composed with a straight line is still a straight line). We need a bend.

Why the topic needs this: The parent stores self.relu = nn.ReLU() and calls self.relu(self.fc1(x)). ReLU is stateless (no dials to learn) but still lives inside the module for a consistent pattern.

5. The function-stack notation $f(x)=f_3(f_2(f_1(x)))$

The parent opens with this. Read it inside-out.

Why the topic needs this: A network's forward() method is literally this line spelled out:

x = self.layer1(x)   # f1
x = self.activation(x)  # bend
x = self.layer2(x)   # f2

Understanding "output of one becomes input of next" is the entire mental model of forward.

6. Parameters vs. inputs — the two kinds of numbers

This is the distinction that motivates nn.Module existing at all.

Why the topic needs this: In the naive function my_layer(x, w, b) you must hand-carry w and b everywhere. nn.Parameter lets the module hold onto its own weights so you never pass them by hand — and so PyTorch can auto-collect them via .parameters().

7. The gradient $\frac{\partial L}{\partial x}$ — the "which way to nudge" arrow

The residual-block section is full of $\frac{\partial L}{\partial x}$. Here is what it means with zero calculus background.

Why the topic needs this: The parent's residual argument $\frac{\partial L}{\partial x}=\frac{\partial L}{\partial \text{out}}\cdot \left(\frac{\partial F(x)}{\partial x}+1\right)$ says the skip connection adds a guaranteed "$+1$" to the slope, so the downhill signal never fades to zero in deep networks. You don't need to compute this — you need to read it as "the $+1$ keeps the training signal alive."

8. The parameter-count arithmetic

The parent asserts 117121 parameters. Now every symbol above lets you verify it.

How the foundations feed the topic

Equipment checklist

Test yourself — say the answer out loud before revealing.