Training loops from scratch: Foundations — AI-ML deep dive

This page assumes nothing. Before you can read the parent note Training loops from scratch, you must be fluent in a small pile of symbols and pictures. We build each one from zero, in an order where every symbol only uses symbols already defined.

1. The knob: a parameter $θ$

The picture. Imagine a mixing desk with hundreds of sliders. Each slider is one number the model can change. $θ$ is the whole desk — the complete list of slider positions right now.

Why the topic needs it. Training does exactly one thing: it changes $θ$ . Everything else — the loss, the gradient, the loop — exists only to decide how to change $θ$ next. If you don't know what $θ$ is, nothing after it makes sense.

Recall

What does $θ$ collect in one symbol? ::: Every weight and bias in the model — the whole "mixing desk" of tunable numbers.

2. The machine: the model function $f (x; θ)$

The semicolon matters. $x$ before the semicolon is the data (changes every example). $θ$ after the semicolon is the knob setting (fixed during a single guess, changed by training). Same machine, two very different roles for what goes in.

The picture. A box. Left arrow in = $x$ . Right arrow out = the guess. The dials on top = $θ$ .

Recall

Why is $θ$ written after a semicolon and $x$ before it? ::: $x$ is the data that varies per example; $θ$ is the knob setting held fixed during one prediction and adjusted by training.

3. Input $x$ , true answer $y$ , guess $\overset{y}{^}$

The hat means "estimated". A plain $y$ is the truth from the real world. A hat $\overset{y}{^}$ is our machine's attempt at that truth. They are rarely equal — the gap between them is the whole point of training.

Why the topic needs it. The distance between the orange guess $\overset{y}{^}$ and the teal truth $y$ is what we shrink. No hat, no gap, no learning.

4. Subscripts and the dataset: $x_{i}$ , $y_{i}$ , $N$

The picture. A spreadsheet with $N$ rows. Row $i$ holds one $(x_{i}, y_{i})$ pair. The subscript is "which row".

Recall

In $x_{7}$ , what does the little $7$ mean? ::: It is the row number — the 7th training example — not a power or multiplication.

5. Summation $\sum$ and the average

Read it left to right: "sum, starting at $i = 1$ , ending at $i = N$ , of ...". The letter under $\sum$ (here $i$ ) is the counter; the number on top ( $N$ ) is where to stop.

Why the topic needs it. The parent note measures error as an average over the dataset. Averaging (not just summing) makes the number independent of dataset size — a loss of "5" means the same thing whether you have 100 or a million examples.

Recall

Why average the loss instead of summing it? ::: A raw sum grows with dataset size; dividing by $N$ makes the loss magnitude comparable across datasets of any size.

6. The scoreboard: per-example loss $ℓ$ and total loss $L (θ)$

We need two loss symbols, not one — a small script $ℓ$ for one example, and a big script $L$ for the whole dataset. Keeping them separate is what makes batches make sense later.

Why squared? The term $(\overset{y}{^}_{i} - y_{i})$ is the miss for one example — it can be positive (guessed too high) or negative (too low). Squaring does two jobs: it makes every miss positive (so misses can't cancel out), and it punishes big misses far more than small ones (a miss of 4 costs 16; a miss of 1 costs only 1).

Now stack the per-example losses over the whole dataset and average them. That gives the total loss, and here is the crucial link: since each $\overset{y}{^}_{i} = f (x_{i}; θ)$ , the total loss is really a function of the knobs $θ$ .

The nested picture $ℓ \circ f$ . Input $x_{i}$ goes into the machine $f$ (which contains $θ$ ), out pops $\overset{y}{^}_{i}$ , which drops into the grader $ℓ$ alongside the truth $y_{i}$ . Loss is $ℓ$ wrapped around $f$ — written $ℓ \circ f$ , read " $ℓ$ after $f$ ".

Why the topic needs it. The total loss is the scoreboard. Without a single $θ$ -dependent number to shrink, "make the model better" has no meaning. It turns a vague wish into a concrete target: find $θ$ that makes $L (θ)$ small.

Recall

What is the difference between $ℓ$ and $L (θ)$ ? ::: $ℓ$ grades a single example; $L (θ)$ is the average of all the $ℓ$ 's and is a function of the knobs $θ$ .

Recall

Why square the errors instead of just adding them raw? ::: Squaring stops positive and negative misses from cancelling, and it penalises large misses much more heavily than small ones.

7. Slope, then the gradient $\nabla_{θ} L$

Everything so far tells us how wrong we are. The gradient tells us which way to turn the knobs.

For a single knob $θ$ , the slope of the loss is written $\frac{\partial L}{\partial θ}$ — read "the change in $L$ per tiny change in $θ$ ". The curly $\partial$ ("partial") just says "we're wiggling one knob while holding the others still".

The picture. On a bowl-shaped loss surface, $\nabla_{θ} L$ is the arrow pointing straight up the steepest wall. The negative of it, $- \nabla_{θ} L$ , points straight downhill toward the bottom of the bowl.

Why this tool and not another? We want to reduce a number ( $L$ ) by changing many knobs ( $θ$ ) at once. The gradient is the only object that answers "which direction, among all possible directions in this high-dimensional knob-space, drops the loss fastest?" The machinery that actually computes it is backpropagation.

Recall

The gradient points uphill on the loss surface. Which way do we step to learn? ::: The opposite direction, $- \nabla_{θ} L$ — downhill, toward smaller loss.

8. The step size $η$ , the iteration index $t$ , and the update rule

The minus sign is the heart of it: subtracting the uphill gradient means we move downhill. Evaluating $\nabla_{θ} L (θ_{t})$ at the current knobs $θ_{t}$ says "measure the steepest-uphill direction from where we're standing right now", then step $t \to t + 1$ .

Why $η$ must be "just right". Too big and you leap over the valley and bounce up the far side (loss explodes). Too small and you inch along forever. Choosing and changing $η$ over time is its own art — see learning-rate schedules.

Recall

Does the counter $t$ tick once per example, once per batch, or once per epoch? ::: Once per batch — i.e. once per parameter update.

Recall

In $θ_{t + 1} = θ_{t} - η \nabla_{θ} L (θ_{t})$ , what does the minus sign accomplish? ::: It flips the "uphill" gradient into a "downhill" move, so the loss decreases.

9. Batch $B_{t}$ , batch size $B$ , one epoch, and the batch update

Computing the gradient over all $N$ examples every single step is slow. So we cheat, cleverly.

The picture. Shuffle the spreadsheet, then slice it into groups of $B$ rows. Process each slice, update the knobs after each. When every row has been used once, that's one epoch. See data loaders and batching for how this slicing is organised.

Instead of averaging $ℓ$ over all $N$ rows, we average it over just the current batch. That gives the batch loss:

Notice this is the same shape as the update rule in §8 — only the whole-dataset loss $L$ has been swapped for the batch loss $L_{B_{t}}$ . That single swap is exactly what makes $t$ tick per batch.

Why batches? A single example gives a noisy, jittery estimate of the true downhill direction; the full dataset gives a perfect direction but is expensive. A mini-batch is the Goldilocks compromise — cheap enough to compute often, stable enough to point roughly downhill. This is the "S" (Stochastic) in SGD.

Recall

What is the batch loss $L_{B_{t}}$ ? ::: The average per-example loss $ℓ$ over just the current batch $B_{t}$ — used to estimate the gradient cheaply.

Recall

What is the difference between a batch and an epoch? ::: A batch is one small handful of examples for a single update; an epoch is one complete pass through all examples, batch by batch.

How these feed the topic

Each box is a symbol you now own. Together they are the training loop: the machine ( $f$ ) guesses ( $\overset{y}{^}$ ), the grader ( $ℓ$ ) scores each guess, the average ( $L (θ)$ or the batch $L_{B_{t}}$ ) becomes the scoreboard, the gradient points the fix, and the update rule turns the knobs ( $θ$ ) by a step $η$ at each tick $t$ — over and over, batch by batch.

Equipment checklist

Test yourself — say the answer aloud before revealing.

What single symbol collects every weight and bias in the model?

θ

What does the semicolon in

f (x; θ)

separate?

The data

x

(before) from the knob settings

θ

(after).

What does the hat in

\overset{y}{^}

signify?

An estimate — the model's prediction, as opposed to the true

y

x_{i}

, is the

i

a power, a multiplier, or an index?

An index — the example's row number.

What does

\sum_{i = 1}^{N}

tell you to do?

Add the following expression for

i = 1

up to

i = N

Why divide the summed loss by

N

To average it, making the loss size independent of dataset size.

What does the per-example loss

ℓ

output?

One number grading a single guess against a single truth.

How is the total loss

L (θ)

built from

ℓ

and

f

L (θ) = \frac{1}{N} \sum_{i} ℓ (f (x_{i}; θ), y_{i})

— the average per-example loss, wrapped around the model

f

, so it depends on

θ

Why square the errors in MSE?

To keep misses positive and to punish large misses more heavily.

What does

\nabla_{θ} L

point toward?

The direction of steepest increase of the loss.

Which way do we step to reduce loss?

Opposite the gradient,

- \nabla_{θ} L

What role does

η

play?

It sets how big each downhill step is (the learning rate).

Does

t

count examples, batches, or epochs?

Batches — it ticks once per parameter update.

What is the batch loss

L_{B_{t}}

and why use it?

The average

ℓ

over the current batch; its gradient is a cheap noisy estimate of the full-dataset gradient.

Next: with every symbol earned, return to Training loops from scratch and watch these pieces click into the five-step cycle. Related paths worth a glance: 3.3.01-PyTorch-fundamentals, 3.3.06-Custom-loss-functions, 3.4.01-Batch-normalization, and 5.1.01-Overfitting-and-underfitting.