Visual walkthrough — TensorFlow - Keras basics

This page takes ONE claim from the parent note — that a neural network is a chain of functions $y=f_n(\dots f_2(f_1(x)))$ — and draws every single arrow of it. By the end you will know exactly what a Dense layer does to numbers, why the softmax at the end is shaped the way it is, and why we normalise pixels before feeding them in.

We assume you know how to multiply two numbers and add them. Nothing else. Every symbol is earned.

Prerequisites we lean on: 3.3.01-Neural-Network-Basics, 3.07-Activation-Functions, 3.1.01-Gradient-Descent, 3.2.03-Backpropagation, 3.4.05-Overfitting-Regularization.

---

Step 1 — A picture is secretly a list of numbers

WHAT. Take one handwritten digit. It is a grid of $28\times 28=784$ tiny squares (pixels). Each square holds one brightness number: $0$ = black, $255$ = white.

WHY. A Dense layer cannot see a 2D grid — it only knows how to work on a flat row of numbers. So we unroll the grid: read it left-to-right, top-to-bottom, into one long list of length $784$. This unrolling is exactly what x_train.reshape(-1, 784) does in the parent note.

PICTURE.

We write this list as a vector — the word "vector" here just means an ordered list of numbers:

$\mathbf{x}=(x_1,x_2,\dots ,x_{784})$

• $\mathbf{x}$ (bold) — the whole list, one image.
• $x_1$ — brightness of the first pixel (top-left corner).
• $784$ — how many pixels, i.e. the length of the list.

---

Step 2 — Rescale the numbers so they all live in $[0,1]$

WHAT. Divide every pixel by $255$. Now the darkest pixel is $0$ and the brightest is $1$; everything else sits between.

WHY. Later steps multiply these numbers by weights. If some inputs are $0$ and others are $255$, the multiplications produce wildly different sizes and the gradient-descent nudges (see 3.1.01-Gradient-Descent) become unbalanced — some directions overshoot, others crawl. Squeezing everything into $[0,1]$ puts all inputs on the same footing, so training converges faster. This is the /255.0 line.

PICTURE.

$x_i⟼\frac{x_i}{255}$

• $x_i$ — original brightness of pixel $i$, somewhere in $0\dots 255$.
• $255$ — the maximum possible brightness (the divisor).
• The arrow $⟼$ — "becomes"; each pixel is replaced by its scaled version.

---

Step 3 — One neuron: multiply, add, sum

WHAT. A single neuron takes the whole input list, gives each entry its own weight, multiplies pixel by weight, adds them all up, then adds one extra number called the bias.

WHY. We want a neuron to be able to say "I care a lot about the pixels in the middle, not at all about the corners." A weight per pixel is precisely a dial for "how much do I care about this pixel." Multiplying and summing lets one number (the neuron's output) reflect a weighted opinion over all $784$ pixels at once.

PICTURE.

For neuron number $j$:

$z_j=\underset{\text{weighted vote of every pixel}}{\underbrace{\sum _{i=1}^{784}{w}_{ij}{x}_i}}+\underset{\text{constant nudge}}{\underbrace{b_j}}$

• $z_j$ — the neuron's raw output (its "pre-activation" score).
• ${\sum }_{i=1}^{784}$ — the "add up over all $784$ pixels" instruction; the big $\Sigma$ is just repeated $+$.
• $w_{ij}$ — the weight connecting pixel $i$ to neuron $j$; how much neuron $j$ cares about pixel $i$.
• $x_i$ — the (rescaled) brightness of pixel $i$.
• $b_j$ — the bias, a fixed offset that shifts the score up or down regardless of the picture.

---

Step 4 — All neurons at once: the matrix–vector multiply

WHAT. A Dense(128) layer has $128$ neurons. Each does the Step-3 computation with its own weights. Stacking all $128$ weighted-sum rules together is exactly matrix multiplication.

WHY. We could run Step 3 in a loop $128$ times. But writing all the weights as one rectangular table $\mathbf{W}$ lets the GPU do every neuron's sum in a single parallel sweep. The maths tool "matrix multiply" is chosen because it packages 128 independent weighted sums into one operation — nothing more mysterious than that.

PICTURE.

$\mathbf{z}=\mathbf{x}\mathbf{W}+\mathbf{b}$

• $\mathbf{x}$ — the input list, length $784$ (one row).
• $\mathbf{W}$ — the weight table, $784$ rows by $128$ columns. Column $j$ holds neuron $j$'s 784 weights.
• $\mathbf{xW}$ — for each column $j$, multiply matching entries and sum → that's $z_j$ from Step 3, all $128$ at once.
• $\mathbf{b}$ — the list of $128$ biases, one per neuron.
• $\mathbf{z}$ — the resulting list of $128$ raw scores.

The parameter count in the parent note follows immediately: $784\times 128$ weights in $\mathbf{W}$ plus $128$ biases $=100,480$. Count it yourself in the figure.

---

Step 5 — Bending the line: the activation function

WHAT. Apply relu to every score: keep positives, replace negatives with $0$.

WHY. Steps 3–4 are purely linear — stretch, tilt, shift. If we stack another linear layer on top, the combination is still just one linear layer (stretching a stretch is a stretch). To recognise curved, tangled shapes like digits we need a bend. relu is the cheapest possible bend: a hinge at $0$. See 3.07-Activation-Functions for the family.

PICTURE.

$a_j=\mathrm{ReLU}(z_j)=\max (0,z_j)$

• $z_j$ — the raw score from Step 4.
• $\max (0,z_j)$ — "pick the larger of $0$ and $z_j$": negatives snap to $0$, positives pass through unchanged.
• $a_j$ — the neuron's activation, the value handed to the next layer.

---

Step 6 — The whole chain: composition of functions

WHAT. Feed $\mathbf{a}$ (the $128$-long bent list) into the next Dense layer, bend again, and so on, exactly as the parent's $y=f_n(\dots f_1(x))$.

WHY. Each layer reshapes the data into a form where the next layer's straight cuts become useful. Early layers might find edges; later layers combine edges into loops and strokes. Composition is how simple bent-lines stack into a curved decision surface.

PICTURE.

$\mathbf{x}\stackrel{f_1}{\to }{\mathbf{a}}^{(1)}\stackrel{f_2}{\to }{\mathbf{a}}^{(2)}\stackrel{f_3}{\to }{\mathbf{z}}^{\text{out}}$

• Each $f_k$ — one "matrix multiply + bias + activation" block (Steps 4–5).
• ${\mathbf{a}}^{(1)}$ — output of layer 1, length $128$.
• ${\mathbf{z}}^{\text{out}}$ — the final $10$ raw scores (one per digit $0-9$), before we turn them into probabilities.

---

Step 7 — Turning $10$ scores into $10$ probabilities: softmax

WHAT. The last layer outputs $10$ raw scores $z_1^{\text{out}},\dots ,z_{10}^{\text{out}}$ (called logits). We convert them into $10$ numbers that are all positive and sum to exactly $1$ — a probability for each digit.

WHY exponential and not something else? We need three properties: (1) every output positive, (2) they sum to 1, (3) a bigger logit stays a bigger probability. The exponential $e^z$ is always positive — that gives (1). Dividing by the total gives (2). Exponential is increasing, so it preserves order — that gives (3). No simpler function hits all three cleanly, which is why softmax uses $e$.

PICTURE.

$\mathrm{softmax}(\mathbf{z}{)}_i=\frac{e^{z_i}}{\sum _{j=1}^{10}{e}^{z_j}}$

• $z_i$ — the raw logit for digit $i$.
• $e^{z_i}$ — exponential; turns any score (even negative) into a positive weight.
• ${\sum }_{j=1}^{10}{e}^{z_j}$ — the total of all ten exponentials; dividing by it forces the outputs to sum to $1$.
• The result — probability that the picture is digit $i$.

---

Step 8 — The degenerate cases (never left guessing)

WHAT & WHY. Three edge cases the reader must be able to survive:

PICTURE.

1. All-black image ($\mathbf{x}=0$). Every weighted sum is $0$, so $z_j=b_j$ — the neuron falls back on its bias alone. The network still produces an output; it just runs on its baseline. (This is why the bias exists.)
2. All logits equal ($z_1=\cdots =z_{10}$). Then every $e^{z_i}$ is the same, so softmax gives $\frac{1}{10}$ to each — a perfectly undecided network. That is the correct "I have no idea" answer, not a crash.
3. relu kills a neuron ($z_j<0$ always). Its activation is stuck at $0$ and it contributes nothing — a "dead neuron." Knowing this explains why people sometimes swap in leaky variants from 3.07-Activation-Functions.

---

The one-picture summary

The entire forward pass, end to end: pixels → flatten → scale → (multiply + bias + bend) → (repeat) → exponentiate & normalise → probabilities. Training (via 3.2.03-Backpropagation and 3.1.01-Gradient-Descent) only adjusts the $\mathbf{W}$ and $\mathbf{b}$ numbers so the correct digit's probability climbs. Everything Keras' model.fit does is hidden inside this one diagram.

Recall Feynman retelling — say it back in plain words

A digit picture is chopped into 784 brightness numbers and shrunk to fit between 0 and 1. Each of 128 neurons multiplies every pixel by its own "how much I care" dial, adds them up, adds a baseline nudge, and then flattens any negative result to zero so the network can bend, not just tilt. Do that a couple more times, and the last step turns ten leftover scores into ten probabilities by exponentiating (to force positivity) and dividing by the total (to force them to add to one). The biggest probability is the network's guess. If the picture is blank, neurons coast on their biases; if all ten scores tie, the answer is an honest one-in-ten shrug. Learning is nothing but slowly turning the "how much I care" dials until the right digit wins.

Recall Quick self-test

What does dividing pixels by 255 fix, and what does it NOT change? ::: It rescales the numbers so gradient descent is well-behaved; it does not change what the image looks like to a human. Why must an activation be non-linear? ::: Otherwise stacking layers collapses into one linear layer — depth buys nothing. Why exponentials in softmax? ::: $e^z$ is always positive (gives valid probabilities) and increasing (preserves which logit is biggest); dividing by the sum makes them add to 1. What happens to softmax when all logits are equal? ::: Every class gets exactly $1/K$ — a uniform "no idea" distribution.

Related framework comparison: 3.3.09-PyTorch-basics. Where this forward pass gets swapped for convolutions: 4.2.01-CNN-Architecture.