Exercises — Chain rule for multivariate functions (backprop foundation)

WHAT: We count paths from $x$ to $z$. There are two: one through $u$, one through $v$. WHY: Each independent route contributes one product of arrow-derivatives; the chain rule sums them. $\frac{\partial z}{\partial x}=\underset{\text{path via }u}{\underbrace{\frac{\partial z}{\partial u}\frac{\partial u}{\partial x}}}+\underset{\text{path via }v}{\underbrace{\frac{\partial z}{\partial v}\frac{\partial v}{\partial x}}}$ Answer: 2 terms, each a product of two arrow-derivatives.

$a$ is linear in each variable. The derivative w.r.t. a variable is just the coefficient sitting next to it. $\frac{\partial a}{\partial w_1}===x==,\frac{\partial a}{\partial x}===w_1==$ WHY: $\frac{\partial }{\partial w_1}(w_{1}x+b_1)=x$ because $x,b_1$ are constants w.r.t. $w_1$.

No. It drops the second path through $v$. Complete answer needs $+\frac{\partial z}{\partial v}\frac{\partial v}{\partial x}$.

Step 1 — arrow-derivatives. $\frac{\partial z}{\partial u}=2uv,\frac{\partial z}{\partial v}=u^2,\frac{\partial u}{\partial x}=1,\frac{\partial v}{\partial x}=y$ Step 2 — sum the two paths ($x\to u\to z$ and $x\to v\to z$). $\frac{\partial z}{\partial x}=2uv\cdot 1+u^2\cdot y=2uv+u^{2}y$ Step 3 — plug in $x=1,y=2$: then $u=3,v=2$. $=2(3)(2)+(3{)}^2(2)=12+18===30==$

Only one path $r\to u\to z$, so we just multiply. $\frac{dz}{dr}=\frac{dz}{du}\frac{du}{dr}=\cos (u)\cdot 6r=6r\cos (3r^2)$ At $r=1$: $u=3$, so $\frac{dz}{dr}=6\cos 3\approx ==-5.9399==$.

WHY this identity matters: it lets backprop reuse the already-computed forward value $h$ instead of recomputing exponentials. ${\sigma }^{\prime }(a)=\frac{e^{-a}}{(1+e^{-a}{)}^2}=\frac{1}{1+e^{-a}}\cdot \frac{e^{-a}}{1+e^{-a}}=\sigma (a)(1-\sigma (a))=h(1-h)$ At $a=0$: $h=\sigma (0)=0.5$, so $\frac{dh}{da}=0.5\times 0.5===0.25==$ (the maximum slope of the sigmoid).

Chain rule (two paths $x\to p\to z$, $x\to q\to z$): $\frac{dz}{dx}=\frac{\partial z}{\partial p}\frac{dp}{dx}+\frac{\partial z}{\partial q}\frac{dq}{dx}=q\cdot e^x+p\cdot 2e^{2x}$ Substitute $p=e^x,q=e^{2x}$: $=e^{2x}{e}^x+e^x\cdot 2e^{2x}=e^{3x}+2e^{3x}=3e^{3x}$ Direct: $z=e^x{e}^{2x}=e^{3x}\Rightarrow \frac{dz}{dx}=3e^{3x}$. ✓ At $x=0$: both give $3e^0===3==$.

Solve $h(1-h)\le 0.01$. Roots of $h-h^2=0.01$: $h=\frac{1\pm \sqrt{1-0.04}}{2}=\frac{1\pm \sqrt{0.96}}{2}$. $\sqrt{0.96}\approx 0.97980\Rightarrow h\le 0.01010\text{or}h\ge 0.98990$ Meaning: when the neuron is saturated (output near $0$ or near $1$), its arrow-derivative is nearly $0$, so its contribution to every gradient is crushed — this is the seed of Vanishing and Exploding Gradients. See also Activation Functions for functions that dodge this. Answer: vanishes for $h\lesssim ==0.0101==$ or $h\gtrsim ==0.9899==$.

$(y-t)=0.8-1=-0.2,h(1-h)=0.6\times 0.4=0.24$ $\frac{\partial L}{\partial w_1}=(-0.2)(2)(0.24)(3)===-0.288==$ WHAT it means: gradient descent updates $w_1\leftarrow w_1-\eta \frac{\partial L}{\partial w_1}$. The gradient is negative, so $-\eta (\text{negative})$ is positive → $w_1$ increases. (See Gradient Descent.)

Forward pass (left to right). $a=0.5(1)+0=0.5,h=\sigma (0.5)=\frac{1}{1+e^{-0.5}}\approx 0.62246$ $y=1\cdot h+0=0.62246,L=\frac{1}{2}(0.62246-0{)}^2\approx 0.19373$ Backward pass (right to left). Reuse each stored value. $\frac{\partial L}{\partial y}=y-t=0.62246$ $\frac{\partial L}{\partial w_2}=\frac{\partial L}{\partial y}\cdot \frac{\partial y}{\partial w_2}=0.62246\cdot h=0.62246\times 0.62246\approx ==0.38746==$ $\frac{\partial L}{\partial b_2}=\frac{\partial L}{\partial y}\cdot 1===0.62246==$ Now push into the hidden layer. $\frac{\partial y}{\partial h}=w_2=1$, $\frac{\partial h}{\partial a}=h(1-h)=0.62246\times 0.37754\approx 0.23500$. $\frac{\partial L}{\partial a}=\frac{\partial L}{\partial y}\cdot w_2\cdot h(1-h)=0.62246\cdot 1\cdot 0.23500\approx 0.14628$ $\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial a}\cdot x=0.14628\times 1\approx ==0.14628==$ $\frac{\partial L}{\partial b_1}=\frac{\partial L}{\partial a}\cdot 1\approx ==0.14628==$ WHY $w_1$ and $b_1$ share $\frac{\partial L}{\partial a}$: both branch off the same node $a$, whose upstream gradient is computed once and reused — that reuse is backprop on the computational graph.

For linear maps the Jacobian is the matrix: $\frac{\partial \mathbf{z}}{\partial \mathbf{u}}=A$, $\frac{\partial \mathbf{u}}{\partial \mathbf{x}}=B$. By the vector chain rule, $\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=AB$ (see Jacobian and Hessian Matrices). $AB=\left[\begin{array}{cc}1& 2\\ 0& 3\end{array}\right]\left[\begin{array}{cc}2& 0\\ 1& 1\end{array}\right]=\left[\begin{array}{cc}1\cdot 2+2\cdot 1& 1\cdot 0+2\cdot 1\\ 0\cdot 2+3\cdot 1& 0\cdot 0+3\cdot 1\end{array}\right]=\left[\begin{array}{cc}==4==& ==2==\\ ==3==& ==3==\end{array}\right]$

Step 1 — sum both output paths into $h$: $\frac{dL}{dh}=\frac{\partial L}{\partial y_1}\frac{d{y}_1}{dh}+\frac{\partial L}{\partial y_2}\frac{d{y}_2}{dh}=1\cdot 1+1\cdot 2h=1+2h$ Step 2 — one more arrow to $x$: $\frac{dh}{dx}=2$. $\frac{dL}{dx}=\frac{dL}{dh}\cdot \frac{dh}{dx}=(1+2h)\cdot 2$ Step 3 — at $x=1$: $h=2$, so $=(1+4)\cdot 2===10==$. This is the "fan-out then fan-in" pattern real graphs use everywhere — see Loss Functions for multi-term losses.

(a) $w_{\text{new}}=w-\eta g=2.0-0.1(0.4)=2.0-0.04===1.96==$. (b) To first order, $\Delta L\approx g\Delta w$. Choosing $\Delta w=-\eta g$ gives $\Delta L\approx g(-\eta g)=-\eta g^2\le 0$. Since $g^2\ge 0$ and $\eta >0$, $L$ can only stay the same (if $g=0$) or decrease. That is the whole justification of Gradient Descent.

${\prod }_{i=1}^{10}0.25=0.25^{10}===9.5367\times 10^{-7}==$ Lesson: each layer shrinks the gradient by $\le 4\times$; after 10 layers the signal reaching early weights is a millionth, so they barely learn. This is the vanishing-gradient problem, motivating ReLU-type Activation Functions whose slope is $1$ (no shrink).

One-path chain rule multiplies; multi-path chain rule ::: multiplies along each path, then adds the paths. Sigmoid's maximum slope $h(1-h)$ occurs at $h=$ ::: $0.5$, giving $0.25$. Backprop reuses each node's upstream gradient how many times? ::: Once, shared to all its children.