Backpropagation algorithm derivation: Question bank

Quick symbol refresher (so no notation is used unexplained):

$z^{(l)}$ = the pre-activation of layer $l$ (the raw weighted sum before squashing).
$a^{(l)} = σ (z^{(l)})$ = the activation (after the nonlinearity $σ$ ).
$δ^{(l)} \equiv \frac{\partial L}{\partial z ^{(l)}}$ = the error signal / "blame" of layer $l$ .
$⊙$ = elementwise (Hadamard) product; $W^{⊤}$ = the transpose of matrix $W$ .

True or false — justify

TRUE or FALSE: Backpropagation is a different rule from the chain rule.

FALSE. It IS the multivariable chain rule — organised so shared sub-derivatives are computed once and cached, rather than a new mathematical law.

TRUE or FALSE: Backprop computes the gradient; it does NOT update the weights.

TRUE. Backprop only produces

\partial L / \partial W

and

\partial L / \partial b

. The actual weight change is a separate step done by Gradient descent (

W \leftarrow W - η \partial L / \partial W

TRUE or FALSE: The backward pass uses the same matrices as the forward pass.

FALSE (in orientation). Forward multiplies by

W^{(l + 1)}

; backward routes blame through the transpose

(W^{(l + 1)})^{⊤}

, because blame flows outputs→inputs, the adjoint direction — the values are shared but the shape is flipped.

TRUE or FALSE: You need one full forward pass before you can do the backward pass.

TRUE. BP2/BP3 require the cached

z^{(l)}

and

a^{(l - 1)}

from the forward sweep; without them

σ^{'} (z^{(l)})

and the outer products are undefined.

TRUE or FALSE: Backprop works only for networks that use the sigmoid activation.

FALSE. The equations hold for any differentiable

σ

; only the specific

σ^{'}

changes. See Activation functions and their derivatives for ReLU, tanh, etc.

TRUE or FALSE:

δ^{(l)}

has the same number of entries as there are neurons in layer

l

TRUE.

δ^{(l)} = \partial L / \partial z^{(l)}

has one component per pre-activation, hence one per neuron in layer

l

TRUE or FALSE: If a layer is purely linear (

σ (z) = z

), backprop still needs the

⊙ σ^{'}

gate.

TRUE — but the gate becomes

σ^{'} (z) = 1

, so it multiplies by one. The gate never disappears; for a linear layer it is simply the identity.

TRUE or FALSE: Making the network deeper always makes the gradient of early weights larger.

FALSE. BP2 multiplies many

σ^{'}

and

W^{⊤}

factors; if these are small the product shrinks toward zero — the vanishing gradient problem. It can grow or shrink.

Spot the error

SPOT THE ERROR: "In BP2 we evaluate the activation derivative as

σ^{'} (a^{(l)})

Wrong argument. It must be

σ^{'} (z^{(l)})

, evaluated at the pre-activation. Sigmoid hides this because

σ^{'} (z) = a (1 - a)

is expressible in

a

; for ReLU using

a

instead of

z

's sign gives wrong gradients.

SPOT THE ERROR: "BP2 reads

δ^{(l)} = W^{(l + 1)} δ^{(l + 1)} ⊙ σ^{'} (z^{(l)})

Missing transpose.

W^{(l + 1)}

is (size

l + 1) \times

(size

l

); to produce a length-

l

vector from a length-

(l + 1)

blame you need

(W^{(l + 1)})^{⊤}

. Dimensions alone rule out the plain

W

SPOT THE ERROR: "

\partial L / \partial W^{(l)} = (a^{(l - 1)})^{⊤} δ^{(l)}

, an inner product giving a scalar."

Order and shape wrong. BP3 is the outer product

δ^{(l)} (a^{(l - 1)})^{⊤}

, a full matrix matching

W^{(l)}

's shape — one gradient per weight, not a single number.

SPOT THE ERROR: "The bias gradient is

δ^{(l)} \cdot a^{(l - 1)}

No. Since

\partial z_{j}^{(l)} / \partial b_{j}^{(l)} = 1

, the bias gradient is just

δ^{(l)}

(BP4). The activation factor belongs to the weight gradient, not the bias.

SPOT THE ERROR: "For MSE

L = \frac{1}{2} (a - y)^{2}

, the output blame

δ^{(L)}

equals

a - y

Incomplete.

\nabla_{a} L = a - y

is only the first factor of BP1; you still multiply by

σ^{'} (z^{(L)})

δ^{(L)} = (a - y) ⊙ σ^{'} (z^{(L)})

. (The gate cancels only for the special softmax + cross-entropy pairing — see Loss functions (MSE, cross-entropy).)

SPOT THE ERROR: "To get

\partial L / \partial W^{(1)}

we chain all the way from

L

back through every layer, freshly each time."

That defeats backprop. The whole point is that

δ^{(1)}

reuses the already-cached

δ^{(2)}, \dots, δ^{(L)}

— dynamic programming, turning exponential recomputation into linear cost.

Why questions

WHY must we sum over

k

in the BP2 derivation?

Because one neuron's output

z_{j}^{(l)}

fans out to every neuron

k

in the next layer. Total sensitivity is the sum of blame over all those forward paths — the multivariable chain rule for a variable that influences many downstream variables.

WHY is the derivative

σ^{'}

called a "gate" on the incoming error?

It scales the blame by how responsive the neuron was: if

σ^{'} (z) \approx 0

(saturated), the neuron barely changed its output, so little blame passes through — the gate is nearly shut.

WHY does backprop cache later-layer deltas instead of recomputing?

The gradient of any early weight contains the later deltas as factors. Computing them once and storing them makes every earlier gradient cheap — the same shared-subproblem trick as autodiff.

WHY does the transpose

W^{⊤}

appear specifically in the backward direction?

Forward,

W

maps inputs→outputs; the sensitivity of an output to an input is the adjoint of that map, which for a matrix is its transpose. Blame travels the reverse arrows of the computational graph.

WHY can't we skip the forward pass and start backward with random activations?

BP2/BP3 depend on the actual

z^{(l)}

and

a^{(l - 1)}

produced by the current inputs and weights. Wrong activations give

σ^{'}

evaluated at the wrong point, so the gradients no longer point downhill on the true loss surface.

WHY do saturating activations like sigmoid cause vanishing gradients in deep nets?

Sigmoid's

σ^{'} (z) = a (1 - a)

peaks at

0.25

and is tiny when saturated. BP2 multiplies one such factor per layer, so a product of many sub-one numbers decays geometrically — see Vanishing and exploding gradients.

WHY is the outer product the right shape for

\partial L / \partial W^{(l)}

Weight

W_{j i}^{(l)}

connects input

i

to neuron

j

, so its gradient is (blame of

j

)

\times

(activity of input

i

)

= δ_{j}^{(l)} a_{i}^{(l - 1)}

. Sweeping all

j, i

pairs is exactly the outer product

δ^{(l)} (a^{(l - 1)})^{⊤}

Edge cases

EDGE CASE: A ReLU neuron sits at

z < 0

(output

0

). What blame passes back through it?

None. ReLU's derivative is

0

for

z < 0

, so the gate shuts and

δ

for that path is zero — a "dead" neuron receives no gradient and stops learning until upstream weights push its

z

positive.

EDGE CASE: All weights in a layer are initialised to the same value. What breaks?

Symmetry. Every neuron in the layer computes identical

z

, gets identical

δ

, and receives identical updates — they stay clones forever. Random init is needed to break this; backprop is correct but useless here.

EDGE CASE: The loss doesn't depend on the network at all (constant loss). What is

δ^{(L)}

Zero.

\nabla_{a} L = 0

, so BP1 gives

δ^{(L)} = 0

, and BP2 propagates zeros backward — every gradient is zero, meaning gradient descent takes no step. Correct behaviour: nothing to learn.

EDGE CASE: For the output layer with softmax activation + cross-entropy loss, why does

δ^{(L)}

simplify to

a^{(L)} - y

The softmax Jacobian and the cross-entropy derivative cancel exactly, leaving

δ^{(L)} = a - y

with no explicit

σ^{'}

gate. This is a special algebraic coincidence, not the general BP1 — see Loss functions (MSE, cross-entropy).

EDGE CASE: A single-layer network (

L = 1

). Do we ever use BP2?

No. With only an output layer, BP1 gives

δ^{(1)} = δ^{(L)}

directly, then BP3/BP4 give the gradients. The recurrence BP2 only fires when there is a next layer to receive blame from.

EDGE CASE: Two forward paths from an early neuron reconverge later (skip connection). Does BP2 still hold?

Yes, if you sum blame over all incoming paths at the reconvergence point. The multivariable chain rule adds contributions from every path; standard BP2 is the special case of a strictly layered graph.

Recall Self-test before you leave

Cover every answer above and re-derive the reason, not just the verdict. Which single factor turns a correct-looking backward equation into a linear-network gradient if you drop it? ::: The $⊙ σ^{'} (z^{(l)})$ nonlinearity gate. Which orientation of $W$ routes blame backward, and why? ::: The transpose $W^{⊤}$ — the adjoint of the forward map, forced also by dimensions.