Neural Network Fundamentals

Question 1 (10)

(a) $\mathbf{z}^{[1]}=W^{[1]}\mathbf{x}+\mathbf{b}^{[1]}$. Row1: $0.5(1)+(-0.5)(2)+0 = 0.5-1 = -0.5$. Row2: $0.3(1)+0.8(2)+1 = 0.3+1.6+1 = 2.9$. $\mathbf{z}^{[1]}=[-0.5, 2.9]^T$. (3: 1 setup, 2 for both values)

(b) $\tanh(-0.5)=-0.4621$, $\tanh(2.9)=0.9940$. $\mathbf{a}^{[1]}=[-0.4621, 0.9940]^T$. (3)

(c) $\hat{y}=W^{[2]}\mathbf{a}^{[1]}+b^{[2]} = 1(-0.4621)+(-2)(0.9940)+0.5$ $= -0.4621 - 1.9880 + 0.5 = -1.9501$. (2)

(d) Bias shifts the activation threshold/decision boundary away from the origin; without it every hyperplane must pass through the origin, so no amount of weight scaling can translate the function. (2)

Question 2 (10)

(a) $\sigma(z)=(1+e^{-z})^{-1}$. $\sigma'(z)= -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \frac{e^{-z}}{(1+e^{-z})^2}$. Write $\frac{e^{-z}}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}} = \sigma(z)(1-\sigma(z))$, since $\frac{e^{-z}}{1+e^{-z}} = 1-\sigma(z)$. (3)

(b) $\text{LReLU}(z)=z$ if $z>0$ else $\alpha z$. Derivative $=1$ if $z>0$, $=\alpha$ if $z<0$. (2)

(c) Sigmoid/tanh derivatives are bounded ($\le 0.25$ for sigmoid, $\le 1$ for tanh) and $\to 0$ in saturation; multiplying many such factors during backprop shrinks gradients toward zero (vanishing). ReLU derivative is exactly 1 on the active region, so gradient magnitude is preserved through many layers. (3)

(d) Exact GELU: $\text{GELU}(z)=z\cdot\Phi(z)$ where $\Phi$ is the standard normal CDF, i.e. $z\cdot\frac{1}{2}\left(1+\text{erf}(z/\sqrt2)\right)$. Advantage: smooth/non-monotonic, non-zero gradient for small negatives (no dead-ReLU units), often better performance. (2)

Question 3 (12)

(a) With $p_i=e^{z_i}/S$, $S=\sum_j e^{z_j}$. $i=k$: $\frac{\partial p_i}{\partial z_k}=p_i(1-p_i)$. $i\neq k$: $\frac{\partial p_i}{\partial z_k}=-p_i p_k$. Compactly $\frac{\partial p_i}{\partial z_k}=p_i(\delta_{ik}-p_k)$. (4)

(b) $\frac{\partial L}{\partial z_k}=-\sum_i y_i\frac{1}{p_i}\frac{\partial p_i}{\partial z_k} = -\sum_i y_i\frac{1}{p_i}p_i(\delta_{ik}-p_k) = -\sum_i y_i(\delta_{ik}-p_k)$ $= -y_k + p_k\sum_i y_i = p_k - y_k$ (since $\sum_i y_i=1$). (5)

(c) $\mathbf{z}=[2,1,0]$: $e^2=7.389, e^1=2.718, e^0=1$, $S=11.107$. $\mathbf{p}=[0.6652, 0.2447, 0.0900]$. True class 0 ⇒ $\mathbf{y}=[1,0,0]$; gradient $=\mathbf{p}-\mathbf{y}=[-0.3348, 0.2447, 0.0900]$. (3)

Question 4 (12)

(a) $\delta^{[2]}=\hat{\mathbf{y}}-\mathbf{y}$ $\frac{\partial L}{\partial W^{[2]}}=\delta^{[2]}(\mathbf{a}^{[1]})^T$ $\frac{\partial L}{\partial \mathbf{b}^{[2]}}=\delta^{[2]}$ $\delta^{[1]}=(W^{[2]T}\delta^{[2]})\odot g'(\mathbf{z}^{[1]})$ $\frac{\partial L}{\partial W^{[1]}}=\delta^{[1]}\mathbf{x}^T$ $\frac{\partial L}{\partial \mathbf{b}^{[1]}}=\delta^{[1]}$ (6)

(b) (row = example; caches A0=X, Z1, A1, Yhat)

def backward(X, Y, W1, b1, W2, A0, Z1, A1, Yhat, g_prime):
    N = X.shape[0]
    dZ2 = (Yhat - Y) / N            # (N, C)
    dW2 = A1.T @ dZ2               # (H, C)
    db2 = dZ2.sum(axis=0)         # (C,)
    dA1 = dZ2 @ W2.T              # (N, H)
    dZ1 = dA1 * g_prime(Z1)       # (N, H)
    dW1 = X.T @ dZ1               # (D, H)
    db1 = dZ1.sum(axis=0)        # (H,)
    return dW1, db1, dW2, db2

(6: correct dZ2 w/ 1/N, matmul orders, summed biases)

Question 5 (10)

(a) Xavier/Glorot: $\text{Var}(W)=\frac{2}{n_{in}+n_{out}}$ (or $\frac1{n_{in}}$), targets tanh/sigmoid. He: $\text{Var}(W)=\frac{2}{n_{in}}$, targets ReLU (accounts for half the units being zeroed). (4)

(b) If per-layer variance grows, signals/gradients explode; if it shrinks, they vanish. Keeping $\text{Var}(a^{[l]})\approx\text{Var}(a^{[l-1]})$ keeps forward activations and backward gradients at stable scale so deep nets train. (3)

(c) With $\mathcal{N}(0,1)$ and large $n_{in}$, pre-activation variance $\approx n_{in}$ grows → large $|z|$ (exploding pre-activations); large $|z|$ pushes sigmoid into saturation where $\sigma'\to0$, so backprop gradients vanish. (3)

Question 6 (6)

(a) Nodes: $u=wx$ (inputs $w,x$) → $v=u+b$ (input $b$) → $f=\sigma(v)$. (2)

(b) Forward: $u=2$, $v=1$, $f=\sigma(1)=0.7311$. Backward: $\frac{\partial f}{\partial v}=f(1-f)=0.7311\cdot0.2689=0.1966$; $\frac{\partial v}{\partial u}=1$; $\frac{\partial u}{\partial w}=x=1$. $\frac{\partial f}{\partial w}=0.1966\cdot1\cdot1=0.1966$. (4)

[
  {"claim":"Q1 output yhat = -1.9501","code":"import math\nz1=[-0.5,2.9]\na=[math.tanh(z1[0]),math.tanh(z1[1])]\nyhat=1*a[0]+(-2)*a[1]+0.5\nresult=abs(yhat-(-1.9501))<1e-3"},
  {"claim":"Q3 softmax gradient = [-0.3348,0.2447,0.0900]","code":"import math\nz=[2,1,0]\ne=[math.exp(v) for v in z]\nS=sum(e)\np=[v/S for v in e]\ny=[1,0,0]\ng=[p[i]-y[i] for i in range(3)]\nres=[round(v,4) for v in g]\nresult=res==[-0.3348,0.2447,0.09]"},
  {"claim":"Softmax-CE gradient identity dL/dz = p - y (symbolic 2-class check)","code":"z0,z1,y0,y1=symbols('z0 z1 y0 y1')\nS=exp(z0)+exp(z1)\np0=exp(z0)/S\np1=exp(z1)/S\nL=-(y0*log(p0)+y1*log(p1))\ncheck=simplify(diff(L,z0)-(p0-y0))\nresult=check==0 or simplify(check.subs({y1:1-y0}))==0"},
  {"claim":"Q6 df/dw = 0.1966","code":"import math\nf=1/(1+math.exp(-1))\ndfdw=f*(1-f)*1*1\nresult=abs(dfdw-0.1966)<1e-3"}
]