Implementing models from scratch: Question bank

Before we start, a shared vocabulary so no symbol appears un-earned:

$x$ = one input vector (a column of numbers, the "features" of one sample).
$W$ = weight matrix, $b$ = bias vector; together they do $z = Wx + b$ (stretch/rotate, then shift).
$σ$ = activation, a bend applied to each number of $z$ so stacked layers aren't secretly one line.
$L$ = loss, a single number scoring "how wrong". $\nabla_{θ} L$ = the gradient, the arrow pointing toward steepest increase of $L$ .
$δ = \frac{\partial L}{\partial z}$ = the gradient that arrives at a layer from the layer after it.
$α$ = learning rate, the step size we take against the gradient.

True or false — justify

A linear layer

z = Wx + b

with no activation can still learn XOR if you stack enough of them.

False. Composing affine maps gives another affine map (

W_{2} (W_{1} x + b_{1}) + b_{2} = W^{'} x + b^{'}

), so a hundred linear layers still draw one straight boundary — XOR needs the bend a nonlinear

σ

provides.

Bias

b

is optional decoration; setting it to zero rarely hurts.

False in general. Without bias every decision boundary is forced through the origin, so a layer cannot shift its threshold — problems whose separating line doesn't pass through zero become unlearnable.

The gradient

\nabla_{θ} L

points in the direction that decreases the loss fastest.

False. It points toward steepest increase; that is exactly why the update subtracts it:

θ_{t + 1} = θ_{t} - α \nabla L

A mini-batch gradient is a worse thing than the full-dataset gradient because it's only an estimate.

Mostly false in practice. It's a noisier estimate, but the noise is unbiased (law of large numbers) and the randomness can help escape sharp minima, all while being far cheaper per step.

Caching the input

X

during the forward pass is a memory optimization we could skip.

False. The weight gradient is

δ X^{T}

— it literally needs

X

. Skipping the cache means recomputing the forward pass during backward, which is slower, not leaner.

ReLU has no gradient at

z = 0

, so training breaks there.

False in practice. The point

z = 0

has measure zero and we just pick a subgradient (usually

0

); training proceeds fine because we essentially never land exactly on the kink.

Momentum makes every step larger, so it always speeds up learning.

False. Momentum accumulates a running direction; on a consistent slope it accelerates, but if gradients keep flipping sign it cancels them and can slow you — it smooths, it doesn't blindly amplify.

Increasing the learning rate

α

always makes convergence faster.

False. The loss-decrease guarantee

L (θ - α \nabla L) \approx L - α ∥\nabla L ∥^{2}

only holds for small

α

; too large and the quadratic terms dominate, so you overshoot and diverge.

The chain rule used in backprop is a different rule from the chain rule in single-variable calculus.

False. It's the same rule, just bookkept with matrices:

\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Z} \frac{\partial Z}{\partial W}

is "outer derivative times inner derivative" done for every element at once.

Spot the error

"For the weight gradient I wrote dW = dL_dZ @ X (no transpose)."

Error: shapes don't align.

δ

(m \times n_{ba t c h})

and

X

(n \times n_{ba t c h})

; you must multiply by

X^{T}

to contract the batch axis and land on

(m \times n)

, the shape of

W

"To send the gradient to the previous layer I use dL_dX = W @ dL_dZ."

Error: forward used

W

to send

X \to Z

, so backward must reverse it with

W^{T}

: dL_dX = W.T @ dL_dZ. Using

W

untransposed mismatches dimensions and reverses no transformation.

"I summed the bias gradient over the feature axis: db = np.sum(dL_dZ, axis=0)."

Error: bias is shared across the batch, not across neurons. You must sum over the batch axis (axis=1, keepdims=True) so each neuron keeps its own accumulated gradient.

"ReLU backward: return dL_dA * (self.A > 0) using the cached output A."

Subtle error: the gate should test the input

z > 0

, not the output. Here they happen to agree because ReLU output is positive exactly when input was, but the clean, general rule is (self.Z > 0).

"My Xavier init is W = np.random.randn(out, in) — I scale the loss instead."

Error: initialization scale and loss scale are different medicines. Xavier divides weights by

in

to keep the variance of activations stable across layers; scaling the loss just rescales gradients and doesn't fix exploding/vanishing signal in the forward pass.

"I update weights inside the loop over the batch, once per sample."

Error: that turns mini-batch SGD into pure online SGD and multiplies your effective learning rate by the batch size. Accumulate/average the gradient over the batch, then take one step.

"I forgot to zero the accumulated gradient before the next batch."

Error: without resetting, gradients from the previous batch add on top, so each step uses a stale, ballooning direction. In from-scratch code you must clear dW/db (or overwrite them) every iteration.

Why questions

Why divide the weight gradient by batch_size in the update?

Because the loss is the mean over the batch, so its gradient is a mean too; dividing keeps the step size independent of how many samples you happened to put in the batch.

Why does the transpose appear in both

δ X^{T}

and

W^{T} δ

In both cases transpose lines up the axis you want to sum over. For

δ X^{T}

it contracts the batch axis; for

W^{T} δ

it maps output-space gradients back into input space, undoing the forward multiply.

Why must the activation derivative be multiplied element-wise (

⊙

) rather than as a matrix product?

Because

σ

acts on each entry independently —

a_{i} = σ (z_{i})

depends only on its own

z_{i}

— so its Jacobian is diagonal, and multiplying by a diagonal is exactly element-wise scaling.

Why is a purely linear network still useful to implement if it can't learn nonlinear boundaries?

It's the cleanest testbed: you can verify your forward shapes, your transpose logic in backprop, and your update loop against closed-form linear algebra before nonlinearities hide any bugs.

Why does momentum use an exponential moving average instead of a simple average of all past gradients?

An EMA needs only one stored vector (velocity) and automatically down-weights ancient, now-irrelevant gradients, whereas a true average would require storing everything and would react too slowly as the landscape changes.

Why can a from-scratch backprop pass every shape check and still train wrong?

Correct shapes don't guarantee correct content: a transpose in the wrong place, a missing activation-derivative multiply, or an un-cached input can all produce right-sized-but-wrong gradients. That's why you verify against finite differences, not just assert shape.

Edge cases

What happens to the gradient through a neuron whose ReLU input was

z = - 2.5

The output is

0

and the local derivative is

0

, so no gradient passes back through that neuron for that sample — it is temporarily "dead" and its upstream weights get no learning signal from this input.

What if an entire neuron's inputs are negative for every sample in the batch?

It stays dead across the whole batch: zero output, zero gradient, zero update, so it never revives on its own — this is the classic dying-ReLU failure that motivates leaky variants or careful initialization.

What does the update do when the gradient is exactly the zero vector?

Nothing moves:

θ_{t + 1} = θ_{t} - α \cdot 0 = θ_{t}

. You've reached a stationary point — could be a minimum, maximum, or saddle, and gradient descent alone can't tell which.

What is the smallest sensible batch size, and what breaks below it?

One (pure SGD) still works mathematically, but the gradient estimate is extremely noisy and any batch-statistic layer (e.g. batch-norm) has no meaningful variance to compute, so those layers degenerate.

What happens as the learning rate

α \to 0

Each step barely moves, so training approaches the smooth gradient-flow trajectory but takes forever — you get stability at the cost of speed, and floating-point underflow can stall progress entirely.

If two samples in a batch produce equal-and-opposite gradients, what does the averaged step do?

They cancel to (near) zero, so the batch takes almost no step even though each sample individually "wants" to move — a reminder that the update follows the mean direction, not any single example.

What if input features are on wildly different scales (one in millions, one near zero)?

The loss surface becomes a stretched valley; the gradient points mostly along the steep axis, so vanilla SGD zig-zags and converges slowly. This is why feature normalization matters even before any fancy optimizer.