Neural Network Fundamentals

Question 1

(a) (6 marks) Softmax: $\hat y_k = \dfrac{e^{z_k}}{\sum_j e^{z_j}}$. Jacobian: $\dfrac{\partial \hat y_k}{\partial z_i}=\hat y_k(\delta_{ki}-\hat y_i)$ (1 mark). $L=-\sum_k y_k\ln\hat y_k \Rightarrow \dfrac{\partial L}{\partial z_i}=-\sum_k \dfrac{y_k}{\hat y_k}\dfrac{\partial\hat y_k}{\partial z_i}$ (2). $=-\sum_k \dfrac{y_k}{\hat y_k}\hat y_k(\delta_{ki}-\hat y_i)=-\sum_k y_k(\delta_{ki}-\hat y_i)=-y_i+\hat y_i\sum_k y_k$ (2). Since $\sum_k y_k=1$ (one-hot): $\dfrac{\partial L}{\partial z_i}=\hat y_i-y_i$, i.e. $\hat y - y$. (1)

(b) (6 marks) Let $\delta^{(2)}=\hat y-y$.

• $\dfrac{\partial L}{\partial W_2}=\delta^{(2)}(a^{(1)})^T$ (outer product) — from $z^{(2)}=W_2 a^{(1)}+b_2$. (2)
• $\dfrac{\partial L}{\partial b_2}=\delta^{(2)}$. (1)
• Backprop: $\dfrac{\partial L}{\partial a^{(1)}}=W_2^T\delta^{(2)}$; then $\delta^{(1)}=\dfrac{\partial L}{\partial z^{(1)}}=\big(W_2^T\delta^{(2)}\big)\odot\phi'(z^{(1)})$. (3)

(c) (8 marks) $\delta^{(2)}=\hat y-y=[0.2,\ 0.3-1,\ 0.5]^T=[0.2,\ -0.7,\ 0.5]^T$. (2) $W_2^T=\begin{bmatrix}1&0&-1\\2&1&1\end{bmatrix}$. $W_2^T\delta^{(2)}=\begin{bmatrix}1\cdot0.2+0\cdot(-0.7)+(-1)(0.5)\\ 2(0.2)+1(-0.7)+1(0.5)\end{bmatrix}=\begin{bmatrix}0.2-0.5\\0.4-0.7+0.5\end{bmatrix}=\begin{bmatrix}-0.3\\0.2\end{bmatrix}$. (3) $\tanh'=1-\tanh^2$; $a^{(1)}=[0,0.8]$ so $(1-(a^{(1)})^2)=[1,\ 1-0.64]=[1,\ 0.36]$. (1) $\delta^{(1)}=[-0.3\cdot1,\ 0.2\cdot0.36]=[-0.3,\ 0.072]^T$. (2)

(d) (4 marks) The network is a DAG of primitive ops. Reverse-mode: one forward pass caches intermediate values; one backward pass propagates $\bar{\cdot}$ (adjoints) from the scalar loss using the chain rule, reusing shared subexpressions — each edge traversed once, so cost $\propto$ forward cost, independent of parameter count (2). Forward-mode propagates one input-direction derivative per pass, needing $O(d)$ passes to get all input sensitivities; since #outputs (scalar loss) $\ll$ #inputs, reverse mode is optimal for gradients (2).

Question 2

(a) (7 marks) $\sigma(z)=(1+e^{-z})^{-1}$, $\sigma'=\dfrac{e^{-z}}{(1+e^{-z})^2}=\sigma\cdot\dfrac{e^{-z}}{1+e^{-z}}=\sigma(1-\sigma)$. (2) Maximize $\sigma(1-\sigma)$: max at $\sigma=1/2$ (i.e. $z=0$), value $=1/4$. (2) Chain of $L$ layers: $\left|\dfrac{\partial L}{\partial z^{(1)}}\right|$ multiplies $L$ factors each $=W\cdot\sigma'$; $|\sigma'|\le1/4$, $|W|\le w$, so bound $(w/4)^L$. (2) Vanishing when $w/4<1\Rightarrow w<4$; gradient $\to0$ exponentially. (1)

(b) (7 marks) (1 mark each function+derivative, ~0.75 explanation)

• ReLU: $f=\max(0,z)$, $f'=\mathbf1[z>0]$. Derivative is 1 for positive → no vanishing on active units; dying ReLU when stuck at $z<0$.
• Leaky ReLU: $f=z$ if $z>0$ else $0.01z$; $f'=1$ or $0.01$. Nonzero negative slope prevents dead units.
• ELU: $f=z$ if $z>0$ else $\alpha(e^z-1)$; $f'=1$ or $\alpha e^z$. Smooth, negative saturation with nonzero gradient near 0, pushes mean activations toward zero.
• GELU: $f=z\,\Phi(z)$ (Gaussian CDF); $f'=\Phi(z)+z\phi(z)$. Smooth, differentiable everywhere, allows small gradients for slightly negative inputs. (mitigation sentences = 3)

(c) (8 marks) $z_i=\sum_j W_{ij}x_j$, independent zero-mean terms: $\mathrm{Var}(z_i)=\sum_{j=1}^{n_\text{in}}\mathrm{Var}(W_{ij}x_j)=n_\text{in}\,\sigma_w^2\sigma_x^2$ (using $\mathrm{Var}(WX)=E[W^2]E[X^2]=\sigma_w^2\sigma_x^2$ for zero-mean indep). (3) To preserve variance ($\mathrm{Var}(z)=\sigma_x^2$): need $n_\text{in}\sigma_w^2=1\Rightarrow \sigma_w^2=1/n_\text{in}$ (Xavier, forward). Balancing forward+backward gives $2/(n_\text{in}+n_\text{out})$. (2) ReLU zeroes ~half the units, halving output variance: effective $\mathrm{Var}(a)\approx\frac12 n_\text{in}\sigma_w^2\sigma_x^2$. To keep it $=\sigma_x^2$ need $\frac12 n_\text{in}\sigma_w^2=1\Rightarrow \sigma_w^2=2/n_\text{in}$ (He). (3)

Question 3

(a) (7 marks) UAT: a feedforward net with a single hidden layer of finitely many neurons and a non-constant, bounded (or non-polynomial) activation can approximate any continuous function on a compact set to arbitrary accuracy in sup norm. (2) Construction of bump $\mathbf1[a\le x\le b]$ with sigmoids (steep gain $c$):

• Hidden unit 1: $h_1=\sigma(c(x-a))$ (rises ~0→1 at $x=a$).
• Hidden unit 2: $h_2=\sigma(c(x-b))$ (rises ~0→1 at $x=b$).
• Output: $o=h_1-h_2\to \mathbf1[a\le x\le b]$ as $c\to\infty$. (3) Bias role: the bias $-ca$ (resp. $-cb$) shifts the step's location to $x=a$ (resp. $b$). Without bias the step is fixed at the origin and the bump cannot be positioned; bias sets the threshold/offset that translates the activation along the input axis. (2)

(b) (7 marks)

import numpy as np
 
def forward(X, W1, b1, W2, b2):
    # X: (N, d); W1: (h, d); b1: (h,); W2: (K, h); b2: (K,)
    Z1 = X @ W1.T + b1            # (N, h)
    A1 = np.maximum(0, Z1)        # ReLU
    Z2 = A1 @ W2.T + b2           # (N, K)
    Z2 = Z2 - Z2.max(axis=1, keepdims=True)   # numerical stability
    E  = np.exp(Z2)
    return E / E.sum(axis=1, keepdims=True)    # softmax probs (N, K)

Marks: correct linear+ReLU (2), correct softmax over class axis (2), stable-softmax subtraction of row max (2), correct shapes/vectorization (1). Stability: subtracting the max makes the largest exponent $0$, avoiding exp overflow; softmax is shift-invariant so the result is unchanged.

[
  {"claim": "delta2 = yhat - y = [0.2, -0.7, 0.5]", "code": "yhat=Matrix([0.2,0.3,0.5]); y=Matrix([0,1,0]); d2=yhat-y; result = (d2==Matrix([0.2,-0.7,0.5]))"},
  {"claim": "W2^T delta2 = [-0.3, 0.2]", "code": "W2=Matrix([[1,2],[0,1],[-1,1]]); d2=Matrix([0.2,-0.7,0.5]); v=W2.T*d2; result = (simplify(v-Matrix([-0.3,0.2]))==zeros(2,1))"},
  {"claim": "delta1 = [-0.3, 0.072]", "code": "W2=Matrix([[1,2],[0,1],[-1,1]]); d2=Matrix([0.2,-0.7,0.5]); a1=Matrix([0,0.8]); v=W2.T*d2; deriv=Matrix([1-a1[0]**2,1-a1[1]**2]); d1=Matrix([v[0]*deriv[0], v[1]*deriv[1]]); result = (simplify(d1-Matrix([-0.3,0.072]))==zeros(2,1))"},
  {"claim": "max of sigma(1-sigma) is 1/4 at sigma=1/2", "code": "s=symbols('s'); f=s*(1-s); crit=solve(diff(f,s),s)[0]; result = (crit==Rational(1,2)) and (f.subs(s,crit)==Rational(1,4))"}
]