Foundations — Saving and loading models (checkpoints)

Before you can understand why a checkpoint saves weights, optimizer state, and an epoch number, you need to know what each of those words means as a picture. This page builds every symbol the parent note uses, starting from absolutely nothing.

1. What is a "parameter"? (the number you tune)

Imagine a machine with thousands of tiny knobs. Each knob is one number. Turn the knobs the right way and the machine's guesses get better. Those knobs are the parameters.

The two families of parameters you'll see:

• Weight — written $\mathbf{W}$ (bold, capital). It scales an input: "how strongly does this input matter?"
• Bias — written $\mathbf{b}$ (bold, lowercase). It shifts the result: "start from here regardless of input."

Look at the figure: each cell of the grid is one weight. The whole grid is $\mathbf{W}$. Saving a checkpoint means writing every one of these cell values to disk.

2. What does one layer compute? (the formula that uses W and b)

A layer takes an input list of numbers $\mathbf{x}$, multiplies by weights, adds the bias:

Read it in plain words:

• $\mathbf{x}$ — the input numbers coming in (e.g. the 784 pixel values of a small image).
• $\mathbf{Wx}$ — combine those inputs, each weighted by how much it matters.
• $+\mathbf{b}$ — shift the total.
• $\mathbf{y}$ — the output numbers going to the next layer.

Why this is the atom of everything: the parent note's nn.Linear(784, 256) is exactly this formula with a $256\times 784$ weight grid $\mathbf{W}$ and a length-256 bias $\mathbf{b}$. When we "save the model," we are saving all the $\mathbf{W}$ and $\mathbf{b}$ from every such layer.

3. What is "loss"? (the score we want to lower)

Picture a landscape where your horizontal position is set by the parameters and your height is the loss. Low ground = good model, high ground = bad model. Training is walking downhill.

The symbol $\mathcal{L}$ (fancy curly L) is just a name for "the error score." We say $\mathcal{L}$ depends on the parameters $\theta$ because moving the parameters moves you across the landscape and changes your height.

4. What is $\theta$? (all the knobs, bundled)

Why invent a bundling symbol? Writing "update ${\mathbf{W}}_1,{\mathbf{b}}_1,{\mathbf{W}}_2,{\mathbf{b}}_2,\dots$" is exhausting. $\theta$ lets us say "update the whole model" in one letter. Your position on the loss landscape is $\theta$; your height is $\mathcal{L}(\theta )$.

5. What is a gradient ${\nabla }_{\theta }\mathcal{L}$? (which way is downhill)

You are standing on the loss hill. You want to step down. Which direction is down?

The upside-down triangle $\nabla$ ("nabla") is just notation meaning "the slope-arrow of." The subscript $\theta$ says "slope with respect to the knobs."

In the figure the red arrow is ${\nabla }_{\theta }\mathcal{L}$ (points uphill). We always step along the green arrow, its negative — downhill.

6. The learning rate $\alpha$ (how big a step)

Knowing the downhill direction isn't enough — how far do you step?

One full downhill step is the core rule of training:

Plain reading: "new knobs = old knobs, moved a little ($\alpha$) in the downhill direction ($-{\nabla }_{\theta }\mathcal{L}$)." The learning rate can even change over training — that whole subject is Learning Rate Schedules.

7. Momentum and velocity ${\mathbf{v}}_t$ (why the past matters)

Plain gradient descent forgets everything except the current slope. But a smart walker builds up speed in a consistently downhill direction — like a ball rolling into a valley.

Reading each piece:

• $\beta$ (beta) — the momentum coefficient, usually $0.9$. It says "keep 90% of my old velocity, mix in 10% of the new slope."
• ${\mathbf{v}}_{t-1}$ — the velocity from the previous step (the accumulated inertia).
• The step now moves along ${\mathbf{v}}_t$, not the raw gradient — so a steady direction builds up speed.

How long to rebuild lost momentum? Roughly $\frac{1}{1-\beta }$ steps. For $\beta =0.9$ that is $\frac{1}{0.1}=10$ steps of wasted stumbling. Adam's $m_t$ and $v_t$ buffers in the parent note are the same idea — extra memory the optimizer carries.

8. Epoch and step $t$ (where you are in the journey)

Picture a bookshelf. Reading one page = one step. Finishing the whole book = one epoch. Re-reading the book = the next epoch.

Why save the epoch number? So "future you" knows to resume at epoch 8 instead of restarting at epoch 0. That's the parent note's start_epoch = checkpoint['epoch'] + 1.

9. Serialization: turning memory into a file

Everything above lives in your computer's RAM (temporary). A file on disk is permanent.

PyTorch's torch.save does this via Python's pickle; TensorFlow writes binary weight files. A state dict (parent note) is just a Python dictionary: {layer name → tensor of numbers}. Saving it = writing that dictionary to disk.

Prerequisite map

Each foundation feeds the next; together they explain why a checkpoint must hold parameters + optimizer state + epoch number, which is the heart of the parent topic. These same foundations power Transfer Learning (reload old weights) and Model Deployment (ship the best checkpoint).

Equipment checklist

Self-test: can you answer each before revealing?