TensorFlow - Keras basics: Exercises — AI-ML deep dive

This page is a self-test ladder. Each problem builds on the TensorFlow - Keras basics note. Work each one on paper before opening its solution. Levels climb from "can you spot it" (L1) to "can you invent it" (L5).

Before we start, one reminder about the two symbols this whole page leans on:

The one formula everything below rests on — the Dense layer forward pass:

Level 1 — Recognition

Exercise 1.1 — Tensor rank

A batch of 32 colour images, each $28 \times 28$ pixels with 3 colour channels, is stored as one tensor. State its shape and its rank (number of axes).

Recall Solution

Shape = == $(32, 28, 28, 3)$ ==. Reading the axes: batch $\to$ height $\to$ width $\to$ channels. Rank = number of axes = == $4$ ==. A rank-0 tensor is a scalar, rank-1 a vector, rank-2 a matrix; four stacked axes make a rank-4 tensor. See prerequisite 3.3.01-Neural-Network-Basics for why images become tensors.

Exercise 1.2 — Name the API

Which model-building style does this code use, and why can it not express a two-input network?

model = keras.Sequential([
    keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1)
])

Recall Solution

It is the Sequential API. A Sequential model is a linear stack: each layer has exactly one input tensor and one output tensor, wired head-to-tail. A two-input network needs a branch (two entry points merging later), which a straight stack cannot represent — that needs the Functional API.

Level 2 — Application

Exercise 2.1 — Parameter count of a Dense layer

A Dense(128, input_shape=(784,)) layer. How many trainable parameters?

Recall Solution

Using $n_{in} \cdot n_{o u t} + n_{o u t}$ with $n_{in} = 784$ , $n_{o u t} = 128$ : $784 \times 128 + 128 = 100352 + 128 = 100480 .$ What we did: counted one weight per input-output wire ( $784 \times 128$ ), then added one bias per output neuron ( $128$ ). Why: every wire and every bias is a number gradient descent will tune.

Exercise 2.2 — Total parameters of a small model

keras.Sequential([
    keras.layers.Dense(64, activation='relu', input_shape=(20,)),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(1)
])

Sum the parameters across all three layers.

Recall Solution

Layer 1: $20 \times 64 + 64 = 1280 + 64 = 1344$
Layer 2: $64 \times 32 + 32 = 2048 + 32 = 2080$
Layer 3: $32 \times 1 + 1 = 32 + 1 = 33$

Total $= 1344 + 2080 + 33 = 3457$ . Why the input of layer 2 is 64: the output width of one layer becomes the input width of the next — that's function composition $f_{2} (f_{1} (x))$ made concrete.

Exercise 2.3 — Softmax by hand

Logits $z = [2, 0, - 1]$ . Compute the softmax probabilities (round to 3 dp) and confirm they sum to 1.

Recall Solution

Softmax is $\frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$ . Why exponential? It forces every output positive and preserves ordering, so we get a valid probability distribution over the classes (see 3.07-Activation-Functions). $e^{2} = 7.389, e^{0} = 1.000, e^{- 1} = 0.368, sum = 8.757$ $p = [\frac{7.389}{8.757}, \frac{1.000}{8.757}, \frac{0.368}{8.757}] = [0.844, 0.114, 0.042]$ Sum $= 0.844 + 0.114 + 0.042 = 1.000$ . ✓ Largest logit ( $2$ ) $\to$ largest probability.

Level 3 — Analysis

Exercise 3.1 — Why divide dropout survivors by $(1 - p)$ ?

During training, dropout with rate $p = 0.2$ zeros 20% of activations and scales survivors by $\frac{1}{1 - p}$ . Show that the expected value of a unit is unchanged. Then explain what would go wrong at test time if we skipped the division.

Recall Solution

Let $x$ be the activation. Under inverted dropout, the output $y$ is: $y = ⎩ ⎨ ⎧ 0 \frac{x}{1 - p} prob p prob 1 - p$ Expected value: $E [y] = p \cdot 0 + (1 - p) \cdot \frac{x}{1 - p} = x .$ With $p = 0.2$ : a surviving unit is scaled by $\frac{1}{0.8} = 1.25$ . Why it matters: at test time dropout is off (all units active). If we had NOT scaled during training, each downstream neuron would suddenly receive $\approx 1/ (1 - p) = 1.25 \times$ the summed signal it was trained on — a distribution shift that wrecks accuracy. Scaling during training keeps train-time and test-time means matched. Connects to 3.4.05-Overfitting-Regularization.

Exercise 3.2 — Sparse vs categorical crossentropy

You have labels stored as integers y = [3, 1, 0] for a 4-class problem. A teammate used CategoricalCrossentropy and got a shape error. Diagnose it and state the two valid fixes.

Recall Solution

CategoricalCrossentropy expects one-hot targets, e.g. label $3 \to [0, 0, 0, 1]$ — shape $(3, 4)$ . But y is shape $(3,)$ integers. Mismatch $\Rightarrow$ error. Fix A: switch the loss to SparseCategoricalCrossentropy, which accepts integer labels directly. Fix B: keep the loss and one-hot the labels: y = keras.utils.to_categorical(y, num_classes=4). Both compute the identical loss value; "sparse" just saves you the one-hot expansion.

Exercise 3.3 — Reading a training curve

Over 5 epochs, training loss falls $0.60 \to 0.10$ while validation loss falls $0.55 \to 0.30$ then rises to $0.40$ on the last epoch. What is happening, and which single Keras argument would let you keep the best model automatically?

Recall Solution

Training loss keeps dropping but validation loss turns upward — the textbook signature of overfitting: the model is memorising training data instead of generalising. Why the gap widens: capacity is being spent on noise the validation set doesn't share. The remedy is early stopping: keras.callbacks.EarlyStopping(monitor='val_loss', restore_best_weights=True). It halts when val_loss stops improving and rolls back to the epoch with lowest validation loss. See 3.4.05-Overfitting-Regularization.

Level 4 — Synthesis

Exercise 4.1 — One gradient-descent step by hand

A single weight $w = 0.5$ has loss $L (w) = (w - 2)^{2}$ . Using learning rate $η = 0.1$ , compute the updated weight after one vanilla gradient-descent step $w \leftarrow w - η \frac{d L}{d w}$ .

Recall Solution

Why the derivative? The derivative $\frac{d L}{d w}$ answers "if I nudge $w$ up a hair, does loss rise or fall, and how steeply?" Moving opposite to it decreases loss (see 3.1.01-Gradient-Descent). $\frac{d L}{d w} = 2 (w - 2) = 2 (0.5 - 2) = 2 (- 1.5) = - 3.$ $w_{n e w} = 0.5 - 0.1 \times (- 3) = 0.5 + 0.3 = 0.8 .$ The negative gradient means loss decreases as $w$ grows, so the step pushes $w$ toward the minimum at $w = 2$ . ✓

Exercise 4.2 — Build a Functional-API skip connection

Write a Functional API model that takes a length-16 input, passes it through Dense(16, relu), then adds the original input to that output (a residual/skip connection), then a final Dense(4). Explain why Sequential cannot do this.

Recall Solution

inputs = keras.Input(shape=(16,))
h = keras.layers.Dense(16, activation='relu')(inputs)
res = keras.layers.add([inputs, h])   # skip connection: same shape (16,) required
outputs = keras.layers.Dense(4)(res)
model = keras.Model(inputs=inputs, outputs=outputs)

Why Sequential fails: the add needs two tensors at once — the raw inputs AND the transformed h. A Sequential stack only ever holds the single "current" tensor and has thrown inputs away by then. The Functional API keeps every named tensor available, so you can reach back and merge. The shapes must match ( $16 = 16$ ) for element-wise addition — this is exactly the pattern ResNet-style CNNs use.

Exercise 4.3 — Subclassing with training-only dropout

In a keras.Model subclass, why must dropout receive training=training in call, and what breaks if you hard-code training=True?

Recall Solution

Dropout must be active during training (to regularise) and off during inference (so predictions are deterministic and use the full network). The training flag Keras passes tells the layer which mode it's in. If you hard-code training=True, dropout stays on at prediction time: every model.predict call randomly zeros 20% of units, so the same input gives different, degraded answers each run. Passing the flag through wires the layer to the framework's mode automatically.

Level 5 — Mastery

Exercise 5.1 — Predict `model.summary()` totals

keras.Sequential([
    keras.layers.Dense(128, activation='relu', input_shape=(784,)),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(10, activation='softmax')
])

Without running it, state the parameter count of each layer and the grand total. (Dropout has zero parameters — explain why.)

Recall Solution

Dense(128): $784 \times 128 + 128 = 100480$
Dropout(0.2): $0$ — dropout has no weights or biases; it only masks activations by a fixed random rule, nothing to learn.
Dense(10): $128 \times 10 + 10 = 1280 + 10 = 1290$

Grand total $= 100480 + 0 + 1290 = 101770$ . This matches the "Total params" line Keras prints — you can now audit any Sequential model by hand.

Exercise 5.2 — Crossentropy loss value by hand

For one sample the true class is index $2$ , and the model's softmax output is $p = [0.1, 0.2, 0.7]$ . Compute the sparse categorical crossentropy loss. Then say what the loss would approach if the model were perfectly confident and correct.

Recall Solution

Crossentropy for a single integer label $c$ is $L = - ln (p_{c})$ — it only looks at the probability assigned to the true class. Why the log? It punishes confident wrong answers harshly ( $ln \to - \infty$ as $p_{c} \to 0$ ) and rewards confident right ones ( $ln 1 = 0$ ). Here $c = 2$ , $p_{2} = 0.7$ : $L = - ln (0.7) \approx 0.357.$ If the model were perfect ( $p_{2} \to 1$ ): $L = - ln (1) = 0$ . The loss floor is exactly zero, reached only at perfect confidence in the right class. Links to 3.2.03-Backpropagation, where this loss is what gets differentiated.

Exercise 5.3 — Design decision under a constraint

You must classify 10 digits but your deployment device only has memory for 10,000 parameters total. Design a valid two-Dense-layer network from a 784-input and prove it fits. (Use a single hidden layer of width $h$ , output width 10.)

Recall Solution

Total params as a function of hidden width $h$ : $P (h) = hidden (784 h + h) + output (10 h + 10) = 785 h + 10 h + 10 = 795 h + 10.$ Require $P (h) \leq 10000$ : $795 h + 10 \leq 10000 \Rightarrow h \leq \frac{9990}{795} = 12.56.$ So the largest safe hidden width is $h = 12$ . Check: $795 \times 12 + 10 = 9540 + 10 = 9550 \leq 10000$ . ✓ ( $h = 13$ gives $10345 > 10000$ , too big.) Design: Dense(12, relu, input_shape=(784,)) then Dense(10, softmax).

Recall Self-check: could you re-derive each without the solution?

Which formula gives a Dense layer's parameter count? ::: $n_{in} \cdot n_{o u t} + n_{o u t}$ Why divide dropout survivors by $1 - p$ ? ::: To keep the expected activation equal to $x$ , so train and test means match. Single-sample crossentropy for integer label $c$ ? ::: $- ln (p_{c})$ Why can't Sequential build a skip connection? ::: It holds only the current tensor and discards earlier ones, so it can't merge two tensors.