Worked examples — TensorFlow - Keras basics

This page is a drill. We take the machinery from TensorFlow / Keras basics and push a number through every kind of situation it can hand you — every shape, every zero, every degenerate case, a word problem, and an exam trap. Nothing here needs code to run; we do the arithmetic by hand so you see what the framework does silently.

Before we start, one promise: every symbol is earned. When we write $\mathbf{Wx}+\mathbf{b}$, $\sigma$, $\text{softmax}$, cross-entropy, or "parameter count", we will have already drawn what it means below.

---

The scenario matrix

Think of this topic as a machine with knobs. Each knob can sit in a normal position, a boundary position, or a broken position. The table lists every knob-class we must cover. Each worked example is tagged with the cell it hits.

#	Case class	The "normal" case	The edge / degenerate case we must also show
A	Tensor shape / rank	matrix (rank 2)	scalar (rank 0), single flattened image (rank 1), a batch dimension
B	Dense forward pass	positive weighted sum	a zero input vector, negative pre-activation
C	ReLU activation	positive $z$ passes through	negative $z$ clipped to 0 (the "dead" side)
D	Softmax	spread-out logits	all-equal logits (max entropy), one huge logit (saturation + numerical stability)
E	Cross-entropy loss	confident-correct (small loss)	confident-wrong (huge loss), the perfect prediction (loss $\to 0$)
F	Dropout scaling	training: scale by $1/(1-p)$	inference: identity (no scaling), $p\to 0$ and $p\to 1$ limits
G	Parameter counting	one Dense layer	a whole Sequential stack, a bias-free layer
H	Gradient step	one Adam/SGD update	learning rate too big (overshoot), gradient = 0 (stuck)
I	Word problem	—	design a model end-to-end from a spec
J	Exam twist	—	shape-mismatch trap, training=True vs False

Prerequisites we lean on: 3.3.01-Neural-Network-Basics, 3.07-Activation-Functions, 3.1.01-Gradient-Descent, 3.2.03-Backpropagation, 3.4.05-Overfitting-Regularization.

---

Building the picture first

Every example below lives on this one diagram: an input vector enters a Dense layer, gets a weighted sum, passes an activation, and (for classification) a softmax turns the numbers into probabilities.

Keep that figure in your head; we point back to it constantly. (Every example below reads the columns of $\mathbf{W}$ as "the wires into neuron $j$" — consistent with the $W_{ij}$ index above.)

---

Example 1 — Case A: shapes, from scalar to image batch

Forecast: guess each shape before reading on. How many numbers total does the image batch hold?

1. Loss 4.2 — no indices needed to reach it. Why this step? Rank counts how many indices you must supply to pick one number; a lone number needs zero. → rank 0, shape ().
2. [1,2,3] — one index picks an element. → rank 1, shape (3,).
3. 2×2 matrix — need a row index and a column index. → rank 2, shape (2,2).
4. One flattened MNIST image — the 28×28 grid is unrolled into a single long list of $28\times 28=784$ numbers; one index picks a pixel. → rank 1, shape (784,). Why this step? A Dense layer multiplies one flat vector by $\mathbf{W}$; flattening is what turns the 2-D image into that vector.
5. Batch of 32 such images — stack the 32 length-784 vectors along a new batch axis in front. → shape (32, 784), rank 2. Why this step? Dense layers eat rows: one row = one example. The batch axis is always first in Keras.

Verify: total numbers in the batch $=32\times 784=25088$. A single flattened image has $784$ pixels, and $784=28^2$. ✓

---

Example 2 — Case B + C: one Dense neuron, zero input and the negative (dead ReLU) side

Forecast: with ReLU = "keep positives, zero out negatives", predict which neuron dies in case (a).

Here $z_j=x_1{W}_{1j}+x_2{W}_{2j}+b_j$ — read the columns of $\mathbf{W}$ as "wires into neuron $j$" (this is exactly the $W_{ij}$ convention from the definition above: column $j$ = weights into neuron $j$; look at the middle of the figure).

1. Case (a), neuron 1: $z_1=1\cdot 2+2\cdot 0+1=3$. Why? Column 1 of $\mathbf{W}$ is $[2,0]$. ReLU$(3)=3$.
2. Case (a), neuron 2: $z_2=1\cdot (-1)+2\cdot 3-4=1$. Column 2 is $[-1,3]$. ReLU$(1)=1$. → output $[3,1]$.
3. Case (b) zero input: the sum vanishes, so $z_j=b_j$. Thus $z=[1,-4]$. Why this matters: with no signal, the bias is the pre-activation — bias sets the neuron's resting level.
4. ReLU$([1,-4])=[1,0]$. Neuron 2 is dead here: negative $z$ → exactly 0. That is the flat left arm of ReLU.

Verify: ReLU$(z)=\max (0,z)$. $\max (0,3)=3$, $\max (0,1)=1$, $\max (0,1)=1$, $\max (0,-4)=0$. ✓ See 3.07-Activation-Functions for why the flat side can permanently kill a neuron.

---

Example 3 — Case D: softmax in three regimes (and the stability trick)

Forecast: for the all-equal case, what must the answer be by symmetry? For the dominant case, how close to 1 does the winner get?

The bars in the figure are the three probability outputs.

1. (a) spread. Compute $e^2\approx 7.389,e^0=1,e^{-1}\approx 0.3679$; sum $\approx 8.757$. → $[0.844,0.114,0.042]$. Why exponential? It is positive and monotone, so it keeps ordering while forcing positivity — the three softmax requirements from the parent note.
2. (b) all-equal. Every $e^5$ is identical, so each fraction is $1/3$. → $[0.333,0.333,0.333]$. Why this step? Equal evidence must give equal probability — softmax respects symmetry; this is the maximum-entropy (most uncertain) output.
3. (c) dominant. $e^{10}\approx 22026$ dwarfs the two $e^0=1$. → $[0.99991,0.000045,0.000045]$. Why show this: softmax saturates — a big logit gap pushes probability toward 1, which flattens gradients (a training concern).
4. Numerical-stability edge case. Imagine logits [1000, 999, 998]. Then $e^{1000}$ overflows to infinity on a computer, and $\infty /\infty$ is NaN. The fix tf.keras uses silently: subtract the max first — replace each $z_i$ by $z_i-\max (\mathbf{z})$. Here that gives [0, -1, -2], all $\le 0$, so every $e^{\cdot }$ is safely in $(0,1]$. Why the answer is unchanged: multiplying numerator and denominator by the same constant $e^{-\max (\mathbf{z})}$ cancels: $\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}=\frac{e^{z_i-\max }}{{\sum }_j{e}^{z_j-\max }}$. Same probabilities, no overflow.

Verify: each vector sums to 1: $0.844+0.114+0.042=1.000$; $3\times 0.333=1.0$; $0.99991+2(0.000045)=1.000$. And the shifted softmax of [1000,999,998] equals the (identical) softmax of [0,-1,-2]. ✓

---

Example 4 — Case E: cross-entropy across correct, wrong, and perfect

Forecast: which case blows up? Which drives loss toward zero?

1. (a) correct. $\mathcal{L}=-\ln (0.90)=0.1054$. Why only $p_{\text{true}}$? The one-hot label is 1 on the true class and 0 elsewhere, so every other term $0\cdot \ln p=0$ drops out — cross-entropy only ever grades the probability you assigned to the right answer.
2. (b) wrong. True-class prob is $0.05$: $\mathcal{L}=-\ln (0.05)=2.996$. Why so large? $-\ln$ explodes as its input $\to 0$; the loss punishes confident mistakes hard. This is the gradient signal that fixes the model.
3. (c) perfect. $\mathcal{L}=-\ln (1)=0$. Why zero? No surprise, no penalty — the limiting best case.

Verify: $-\ln 0.9\approx 0.10536$, $-\ln 0.05\approx 2.9957$, $-\ln 1=0$. Wrong loss $\gg$ correct loss $>$ perfect. ✓ (These are the numbers SparseCategoricalCrossentropy() returns per example.)

---

Example 5 — Case F: dropout scaling, train vs inference, and the limits

Forecast: why does the kept value get bigger, not stay the same, during training?

Rule: training keeps a unit with prob $1-p$ and rescales it by $\frac{1}{1-p}$; drops otherwise. Inference does nothing.

1. Training draw (drop units 2,3): survivors scale by $\frac{1}{1-0.5}=2$. → $[4\cdot 2,0,0,6\cdot 2]=[8,0,0,12]$. Why scale up by 2? Half the inputs vanished, so to keep the expected total unchanged we double the survivors.
2. Expected value check (the "why"): $\mathbb{E}[y_i]=(1-p)\cdot \frac{x_i}{1-p}+p\cdot 0=x_i$. For $x_1=4$: $\mathbb{E}=0.5\cdot 8+0.5\cdot 0=4$. Mean is preserved — that is the whole point of inverted dropout.
3. Inference: training=False, so dropout is the identity → output is exactly $[4,8,2,6]$. Why: we want deterministic, full-signal predictions at test time.
4. Limits: $p\to 0$ → scale $\to 1$, nothing dropped (dropout off). $p\to 1$ → scale $\to \infty$ and almost everything zeroed (network destroyed — useless). Sane $p$ lives in $[0.2,0.5]$.

Verify: kept value $4\cdot \frac{1}{1-0.5}=8$; $\mathbb{E}[y_1]=0.5\cdot 8=4=x_1$. ✓ More on this in 3.4.05-Overfitting-Regularization.

---

Example 6 — Case G: counting parameters of a whole Sequential stack

Forecast: does Dropout add any parameters? Guess the total order of magnitude.

A Dense layer with $n_{\text{in}}$ inputs and $n_{\text{out}}$ outputs has $n_{\text{in}}\cdot n_{\text{out}}$ weights $+n_{\text{out}}$ biases.

1. Layer 1: $784\cdot 128+128=100352+128=100480$. Why $+128$? one bias per output neuron.
2. Dropout: $0$ parameters. Why? It only masks/scales; there is nothing to learn.
3. Layer 2: $128\cdot 64+64=8192+64=8256$.
4. Layer 3: $64\cdot 10+10=640+10=650$.
5. Total: $100480+0+8256+650=109386$.
6. Bias-free variant of layer 1: drop its 128 biases → $100480-128=100352$, so grand total $=109258$.

Verify: $784\cdot 128=100352$; sum $100480+8256+650=109386$; bias-free total $109386-128=109258$. ✓ This matches model.summary()'s "Total params".

---

Example 7 — Case H: one gradient-descent update, and the two failure modes

Forecast: does one small step move $w$ toward 5? Does the big step help or hurt?

The derivative $\frac{d\mathcal{L}}{dw}=2(w-5)$ tells us the slope: negative slope means "go right to descend."

1. Gradient at $w=2$: $2(2-5)=-6$. Why the minus sign matters: gradient points uphill; we step opposite it.
2. Good step ($\eta =0.1$): $w\leftarrow 2-0.1\cdot (-6)=2+0.6=2.6$. Loss went $9\to (2.6-5{)}^2=5.76$. Closer to 5. ✓
3. Overshoot ($\eta =1.5$): $w\leftarrow 2-1.5\cdot (-6)=2+9=11$. Loss went $9\to 36$ — worse. Why: too-big $\eta$ jumps past the valley and climbs the far wall; this is divergence.
4. At the minimum ($w=5$): gradient $=2(5-5)=0$, so the update leaves $w$ unchanged. Why: zero slope = flat ground = nowhere to descend. (Real Adam behaves richer, but the SGD skeleton is this.)

Verify: step-down: $2-0.1(-6)=2.6$; loss $5.76<9$. Overshoot: $2-1.5(-6)=11$; loss $36>9$. Minimum: gradient $=0$. ✓ Foundations in 3.1.01-Gradient-Descent and 3.2.03-Backpropagation.

---

Example 8 — Case I: word problem, design the whole model

Forecast: why is the output activation not softmax here?

1. Input shape (20,). One row per customer, 20 numbers wide.
2. Hidden: two ReLU Dense layers, 16 then 8 units — non-linearity so stacked layers don't collapse into one linear map.
3. Output: Dense(1, activation='sigmoid'). Why sigmoid not softmax? We need one probability, not a distribution over classes. Sigmoid squashes any real number into $(0,1)$ — perfect for "probability of the single event default."
4. Loss: BinaryCrossentropy — the two-outcome sibling of the loss in Example 4.
5. Param count: $(20\cdot 16+16)+(16\cdot 8+8)+(8\cdot 1+1)=336+136+9=481$.

Verify: $20\cdot 16+16=336$; $16\cdot 8+8=136$; $8\cdot 1+1=9$; total $481$. A sigmoid output in $(0,1)$ is a valid probability. ✓

---

Example 9 — Case J: two exam traps (shape mismatch & the training flag)

Forecast: how many numbers does one MNIST image have, and how many does the Dense layer expect per row?

Trap (a) — the shape mismatch:

1. What the layer expects. Dense(128, input_shape=(784,)) builds a weight matrix $\mathbf{W}\in {\mathbb{R}}^{784\times 128}$, so it wants each example to be a flat length-784 row. Why: a Dense layer computes $\mathbf{xW}$; that matrix product only lines up if $\mathbf{x}$ has 784 entries.
2. What you actually fed. The raw tensor (32, 28, 28) gives each example as a rank-2 28×28 block, not a flat vector. Why it fails: Keras raises a shape/rank error — it cannot multiply a 2-D block by a 2-D weight matrix along a length-784 axis that isn't there.
3. The fix. Insert keras.layers.Flatten() as the first layer (or reshape to (32, 784) yourself). This unrolls each 28×28 image into length $28\times 28=784$ — exactly Example 1, step 4. Why nothing is lost: flattening only relabels the same 784 pixels as one long row; no pixel is dropped.

Trap (b) — training=True vs False:

4. What causes the randomness. Dropout (Example 5) randomly zeros units — but only when it runs in training mode. If your custom call hard-codes self.dropout(x, training=True), dropout keeps sampling a fresh random mask on every forward pass, so the same input gives different outputs. Why: you asked it to drop units even at test time.
5. The fix. Let the training flag flow through: self.dropout(x, training=training), and call the model with training=False at inference (which model.evaluate / model.predict do automatically). With training=False, dropout's keep-probability is $1$, so it becomes the identity — deterministic, full-signal outputs. Why this is correct: at test time we want the model's honest, stable prediction, not a randomly thinned one.

Verify: $28\times 28=784$ (the flatten fix is exact); with training=False, dropout keep-prob $=1$ and scale $=1$, so it is the identity → deterministic output. ✓

---

Recall Self-check (reveal after answering)

Softmax of all-equal logits [5,5,5]? ::: $[\frac{1}{3},\frac{1}{3},\frac{1}{3}]$ — equal evidence, equal probability. Cross-entropy loss for a perfect prediction on the true class? ::: $-\ln 1=0$. Params in Dense(64) fed by 128 inputs? ::: $128\cdot 64+64=8256$. Why divide by $1-p$ in dropout? ::: to keep $\mathbb{E}[y_i]=x_i$ (preserve expected activation). Output activation for a single-probability default model? ::: sigmoid, not softmax. ReLU of a $z=[1,-4]$ pre-activation? ::: $[1,0]$ — the negative side is clipped to zero. Why subtract $\max (\mathbf{z})$ before softmax? ::: to avoid $e^{\text{large}}$ overflow; the probabilities are unchanged.

Related build-outs: 3.3.09-PyTorch-basics (same math, different steering wheel) and 4.2.01-CNN-Architecture (where the Flatten trap in Example 9 gets replaced by convolutions).