Foundations — TensorFlow - Keras basics

Before you can read one line of the parent note, you need to recognise every symbol and word it throws at you. We build them from nothing, in an order where each rung of the ladder rests on the one below. Everything here supports TensorFlow - Keras basics.

0. What is a number-container? (the tensor)

The parent note's very first sentence talks about a "tensor". Let's earn that word.

Look at the figure. Reading left to right:

• A single number, like $42$, extends in zero directions. Rank 0 — we call it a scalar.
• A row of numbers $[1,2,3]$ extends in one direction. Rank 1 — a vector.
• A grid of rows and columns extends in two directions. Rank 2 — a matrix.
• Stack matrices behind each other and you get rank 3, 4, ... — a colour image is rank 3 (height × width × colour), and a batch of images is rank 4.

Notation — shape. The shape is just the list of how many entries lie along each direction, written in round brackets. In plain words:

• a scalar has an empty shape, written $()$ — no directions.
• a vector of 3 has shape $(3,)$ — one direction, holding 3 numbers.
• a 2 by 2 matrix has shape $(2,2)$ — two directions, 2 rows and 2 columns.
• 32 grey images have shape $(32,28,28,1)$ — four directions.

The comma in $(3,)$ is not a typo — it tells you "one direction, and it holds 3 numbers".

1. The two operations you must recognise: $+$ and matrix-multiply

The parent uses a + b, a * b, and tf.matmul. Two of these are the same everyday idea; one is new.

Element-wise ($+$ and $\ast$): line the two boxes up, act on matching cells.

• Add: $[[1,2],[3,4]]+[[5,6],[7,8]]=[[6,8],[10,12]]$.
• Multiply: $[[1,2],[3,4]]\ast [[5,6],[7,8]]=[[5,12],[21,32]]$ — pair matching cells and multiply ($1\times 5,2\times 6,3\times 7,4\times 8$). This is what a * b means; it is not matrix multiply.

Matrix multiplication (written tf.matmul): this is the new one, and it is the heart of every layer.

For a single number in the output:

Here ${\sum }_{k=1}^K$ means "add these up as $k$ runs $1,2,\dots ,K$", where $K$ is the shared dimension: the number of columns of the left matrix, which must equal the number of rows of the right matrix. Look at the red highlighted row-and-column in the figure; the red output cell is their paired-and-summed value.

2. From symbols to a neuron: $\mathbf{z}=\mathbf{xW}+\mathbf{b}$

Now every symbol in the parent's key formula is within reach.

Decode it piece by piece. The convention Keras uses is row vectors: one example is a horizontal strip of numbers, and a batch stacks such rows on top of each other.

• $\mathbf{x}$ — the input, written as a row vector of shape $(1,n)$ for one example (or $(\text{batch},n)$ for many). With $n=784$ pixels that is a $1\times 784$ strip.
• $\mathbf{W}$ — the weight matrix of shape $(n,m)$: $n$ inputs down, $m$ neurons across. One dial per (input, neuron) pair.
• $\mathbf{b}$ — the bias, a row vector of shape $(m,)$: one extra dial per neuron that shifts its output up or down even when all inputs are zero. Picture the "+c" that lets a line not pass through the origin. It broadcasts across every row of the batch.
• $\mathbf{z}$ — the weighted sum ("logit"), shape $(1,m)$: one number per neuron, before any bend.
• $\sigma$ (Greek "sigma") — the activation function: a bend applied to each number. More on it next.
• $\mathbf{a}$ — the activation: $\mathbf{z}$ after the bend; the neuron's final output, also shape $(1,m)$.

Notice the shapes line up exactly by the rule from Section 1: $(1,\boxed{n})\times (\boxed{n},m)\to (1,m)$, then adding $\mathbf{b}$ of shape $(m,)$ broadcasts fine. That is why Keras writes $\mathbf{xW}$ (row on the left) rather than $\mathbf{Wx}$ — the row-vector convention makes the batch dimension come first and the shapes chain cleanly.

The figure shows three inputs feeding one neuron: each arrow carries a weight $w$, the summing circle does $\mathbf{xW}+\mathbf{b}$, and the red box is the bend $\sigma$.

$\sigma$ cases you'll meet:

• ReLU: $\sigma (z)=\max (0,z)$ — keeps positives, flattens negatives to 0. (The bend at zero.)
• Softmax (whole layer): $\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$ — turns a vector of $K$ raw scores into positive numbers that sum to $1$, i.e. probabilities. Here $e$ is Euler's number, the fixed constant $e\approx 2.718$, and $e^z$ is the exponential function (constant $e$ raised to the power $z$). We use it because $e^z$ is always positive and always grows as $z$ grows, so a bigger score gives a bigger probability, never a negative one.

The single neuron formula generalises to a full layer by letting $\mathbf{W}$ have one column per neuron — that is the matrix multiply from Section 1.

3. Measuring "how wrong": the loss $\mathcal{L}$

The parent writes $\mathcal{L}(\theta )$ and calls it the loss.

Picture $\mathcal{L}$ as the height of a landscape, and $\theta$ as your position on the ground. Training = walking downhill to the lowest valley.

4. The dial-nudge: gradient $\nabla$ and the update rule

The parent's central training line is:

$\theta \leftarrow \theta -\eta {\nabla }_{\theta }\mathcal{L}$

Three new symbols:

• ${\nabla }_{\theta }\mathcal{L}$ — the gradient (nabla). It is the arrow pointing in the direction where the loss-landscape rises fastest. Walking the opposite way (the minus sign) goes downhill.
• $\eta$ (Greek "eta") — the learning rate: how big a step you take. Too big → you leap over the valley; too small → you crawl.
• $\leftarrow$ — "becomes": replace the old dials with the new ones.

The red arrow in the figure is $\nabla \mathcal{L}$ pointing uphill; the black step goes the other way, downhill, shrinking the loss. This is the whole of 3.1.01-Gradient-Descent, and the machinery that computes $\nabla \mathcal{L}$ automatically is 3.2.03-Backpropagation — the reason we let TensorFlow, not our pencil, do the calculus.

5. Words around the training loop

Small vocabulary the parent's model.fit example assumes:

• Batch / batch_size=32: process 32 examples together before nudging the dials once. Fewer nudges, steadier direction.
• Epoch / epochs=5: one full sweep through all the training data. Five epochs = five sweeps.
• Normalize / /255.0: pixels run $0$–$255$; dividing squeezes them to $0$–$1$ so gradients stay well-scaled.
• Validation split: hide 20% of the data during training and test on it, to catch overfitting — memorising instead of learning (see 3.4.05-Overfitting-Regularization).
• Dropout: randomly switch off some activations while training so the network can't lean on any one neuron — also an anti-overfitting trick.

Prerequisite map

The map reads bottom-up: boxes of numbers become neurons, neurons stack into a network, the network's error is measured by a loss, the gradient of that loss drives the update, and all of it is what the parent topic ties together.

Equipment checklist

Cover the right side and answer aloud; reveal to check.