Question bank — Backpropagation — chain rule, gradient computation

False — backprop only computes the gradients; Gradient Descent is the separate step that uses those gradients to update weights. One supplies, the other consumes.

True — the gradient $\partial L/\partial w$ only makes sense when $L$ is a single number; a vector output must first be reduced to a scalar loss before any backward sweep can start.

False — backprop needs the cached activations $a^{(\ell -1)}$ and pre-activations $z^{(\ell )}$ from the forward pass, since local derivatives like ${\sigma }^{\prime }(z)$ and the input factor depend on them.

True — cost scales with the number of operations in the computation graph, so it grows linearly with depth, not with the number of weights individually.

True — the error is multiplied by ${\sigma }^{\prime }(z)=0$ when crossing the nonlinearity, so that path contributes zero to upstream weight gradients (this is one root of Vanishing Gradients).

True — it is the chain rule applied symbolically to the graph, so (up to floating-point) it is exact, unlike finite-difference estimates which carry truncation error.

True — since $z=Wa+b$ gives $\partial z/\partial b=1$, the bias inherits the raw $\delta$ with no input factor attached.

False — automatic differentiation applies exactly these rules to arbitrarily large graphs mechanically; the hand derivation just reveals what the machine does.

It drops the downstream error: $\partial L/\partial w_2={\delta }_2{a}_1$, not $a_1$ alone. The bare $a_1$ is only the local derivative $\partial z_2/\partial w_2$; the rest of the network's blame ${\delta }_2$ multiplies it.

Backprop uses the transpose $W^{\top }$, not the inverse. The derivative of the linear map $Wa$ with respect to $a$ is $W^{\top }$; differentiation transposes, it never inverts (and $W$ need not even be square).

It forgets the $\odot {\sigma }^{\prime }(z^{(\ell )})$ factor. Hidden layers pass through a nonlinearity, so their incoming error must be scaled by each unit's activation slope; only the linear output layer skips it.

You must sum the contributions from all outgoing paths (multivariable Chain Rule). Each path is an independent way the node influences the loss, and total sensitivity is their sum.

For MSE $L=\frac{1}{2}(\hat{y}-t{)}^2$ the seed is $\hat{y}-t$, the error, not $\hat{y}$. Missing the target $t$ makes the whole downstream gradient wrong.

Recomputing each local derivative from scratch would destroy the efficiency; the whole $O(1)$-forward-pass advantage comes from caching activations so each backward step is cheap.

The orientation is wrong — it is the outer product ${\delta }^{(\ell )}{a}^{(\ell -1)\top }$, giving a matrix shaped like $W^{(\ell )}$ (outputs × inputs). Swapping the transpose gives the wrong shape.

Because the chain rule factors the derivative into a product of local slopes, and a single backward sweep reuses each cached intermediate to fill in all weight gradients simultaneously.

The loss is a scalar at the far end; sweeping backward from that one scalar reuses each shared sub-derivative, whereas forward accumulation would recompute the same downstream factors once per input weight.

We are differentiating, so we need the slope of the activation (${\sigma }^{\prime }$), which says how a small change in $z$ changes $a$ — the value $\sigma (z)$ itself is what the forward pass used, not the sensitivity.

Because each layer's pre-activation has the identical linear form $z=Wa+b$, so $\partial z/\partial W$ always picks out the input $a$ — the recursion just repeats this local structure at every depth.

In their flat regions ${\sigma }^{\prime }(z)\approx 0$, so correct math still multiplies the error by near-zero, shrinking gradients across many layers — the mechanism behind Vanishing Gradients.

Fitting a CFD surrogate means tuning millions of weights on huge datasets; only the one-sweep gradient of backprop makes each training step affordable enough to converge.

The slope is undefined there (a kink), so implementations pick a subgradient — usually ${\sigma }^{\prime }(0)=0$ (sometimes $1$); it is a measure-zero case that rarely affects training.

No — the gradient depends on the error arriving and the input at each weight's own location, which generally differ, so equal weights can have very different gradients.

Its ${\sigma }^{\prime }(z)=0$ on all inputs, so it passes zero error backward and receives zero weight gradient forever — it is stuck and cannot recover on its own.

Yes — its gradient is exactly $\delta$, the raw error at that unit; the constant input just means there's no extra input factor multiplying it.

It returns all-zero gradients — a correct answer meaning descent will not move, which happens at minima, maxima, saddle points, or dead-flat plateaus alike.

Zero, regardless of the weight's own value or its input, because $\partial L/\partial w=\delta \cdot a$ and $\delta =0$ nullifies the product.