Neural Network Fundamentals

Question 1 (12 marks)

(a) Pre-activations $z^{(1)} = W^{(1)}\mathbf{x} + \mathbf{b}^{(1)}$:

• Unit 1: $0.5(1) + (-0.5)(-2) + 0 = 0.5 + 1.0 = 1.5$ (1)
• Unit 2: $1.0(1) + 0.0(-2) + 0.5 = 1.5$ (1)

Activations $a^{(1)} = \tanh(z^{(1)})$:

• $\tanh(1.5) = 0.90515$ (1.5)
• $\tanh(1.5) = 0.90515$ (1.5)

So $\mathbf{a}^{(1)} = [0.905, 0.905]^\top$.

(b) Output pre-activation: $z^{(2)} = 1.0(0.90515) + (-1.0)(0.90515) + 0.2 = 0.2$ (2) $\hat{y} = \sigma(0.2) = \frac{1}{1+e^{-0.2}} = 0.5498 \approx 0.550$ (2)

(c) (3 marks, 1.5 each)

• Advantage: ReLU has gradient 1 for positive inputs (here $z=1.5>0$), avoiding tanh's gradient shrinkage ($\tanh'(1.5)\approx0.18$), so it propagates larger gradients → mitigates vanishing. (1.5)
• Disadvantage: ReLU can produce dead units if pre-activations go negative (zero gradient), and it is unbounded, potentially causing exploding activations; tanh is bounded and zero-centered. (1.5)

Question 2 (12 marks)

(a) Softmax: $p_i = e^{z_i}/\sum_j e^{z_j}$.

• $e^{2.0}=7.389,\; e^{1.0}=2.718,\; e^{0.1}=1.105$; sum $=11.212$ (2)
• $p_0 = 7.389/11.212 = 0.659$
• $p_1 = 2.718/11.212 = 0.242$
• $p_2 = 1.105/11.212 = 0.099$ (2)

(b) $L = -\ln p_0 = -\ln(0.659) = 0.417$ (3)

(c) For softmax + cross-entropy the gradient simplifies: $\frac{\partial L}{\partial z_i} = p_i - y_i$ Derivation: $L=-\sum_k y_k \ln p_k$; using $\partial p_k/\partial z_i = p_k(\delta_{ki}-p_i)$ and $\sum_k y_k = 1$ gives $p_i - y_i$. (2)

Values: (3)

• $\partial L/\partial z_0 = 0.659 - 1 = -0.341$
• $\partial L/\partial z_1 = 0.242 - 0 = 0.242$
• $\partial L/\partial z_2 = 0.099 - 0 = 0.099$

Question 3 (11 marks)

(a) Each layer contributes Jacobian factor $\sigma'(z)\cdot w \approx 0.2w$. Over 50 layers: $\left|\frac{\partial L}{\partial a^{(0)}}\right| \propto \prod_{l=1}^{50} |0.2\,w| = |0.2w|^{50}$ (2) For stability (neither vanish nor explode) we need $|0.2w| \approx 1$, i.e. $|w| \approx 5$. If $|0.2w|<1$ → vanish; if $>1$ → explode. (2)

(b) He init: $\text{Var}(w) = 2/n_{in} = 2/256 = 0.0078125$. (2) Std $= \sqrt{0.0078125} = 0.08839$. (1) Reason: ReLU zeroes half its inputs, halving the variance passed forward; the factor 2 compensates to keep signal variance constant. (1)

(c) (3 marks)

• Phenomenon: vanishing gradients (1). Sigmoid derivative $\le 0.25$ compounds multiplicatively over depth. (1)
• Fixes (any two, 1 each): use ReLU/variants; use He/Xavier initialization; add residual/skip connections; batch normalization; LSTM-style gating (if recurrent). (2)

Question 4 (8 marks)

(a) Any two (2 each):

• UAT is an existence result: it guarantees a network exists but says nothing about the number of hidden units, which may be exponentially large for shallow nets. Deep nets can represent many functions far more compactly (parameter efficiency). (2)
• UAT says nothing about learnability — whether SGD can find those weights, or about generalization. Depth aids optimization and hierarchical feature reuse. (2)

(b) Note $|x| = \text{ReLU}(x) + \text{ReLU}(-x)$. (2) Construction:

• Hidden unit 1: $w_1=1, b_1=0 \Rightarrow h_1=\text{ReLU}(x)$
• Hidden unit 2: $w_2=-1, b_2=0 \Rightarrow h_2=\text{ReLU}(-x)$
• Output: $\hat{f} = 1\cdot h_1 + 1\cdot h_2 + 0 = \text{ReLU}(x)+\text{ReLU}(-x) = |x|$ (2)

Verification: for $x>0$, output $=x$; for $x<0$, output $=-x$; at $0$, output $=0$. Exact.

Question 5 (7 marks)

(a) Graph nodes (2 marks): $u = x\cdot y$; $s=\sin x$; $v = u+s$; $f = v^2$. Leaves $x,y$.

(b) Forward: $u=2.0$, $s=\sin 1=0.8415$, $v=2.8415$, $f=8.074$. (1)

Reverse pass:

• $\partial f/\partial v = 2v = 5.6829$ (1)
• $\partial v/\partial u = 1,\; \partial v/\partial s = 1$
• $\partial u/\partial x = y = 2,\; \partial u/\partial y = x = 1$
• $\partial s/\partial x = \cos x = 0.5403$

Combine: (3) $\frac{\partial f}{\partial x} = 5.6829(1\cdot 2 + 1\cdot 0.5403) = 5.6829 \times 2.5403 = 14.437$ $\frac{\partial f}{\partial y} = 5.6829(1 \cdot x) = 5.6829 \times 1 = 5.683$

[
  {"claim":"Q1a hidden pre-activations both 1.5 and tanh(1.5)=0.905","code":"import sympy as sp\nz1=sp.Rational(1,2)*1+sp.Rational(-1,2)*(-2)+0\nz2=1*1+0*(-2)+sp.Rational(1,2)\na=sp.tanh(1.5)\nresult = (z1==sp.Rational(3,2)) and (z2==sp.Rational(3,2)) and abs(float(a)-0.905)<0.001"},
  {"claim":"Q1b output sigmoid(0.2)=0.550","code":"z2=0.2\nout=1/(1+sp.exp(-z2))\nresult = abs(float(out)-0.550)<0.001"},
  {"claim":"Q2 softmax p0=0.659, loss=0.417, grad0=-0.341","code":"import sympy as sp\nz=[2.0,1.0,0.1]\nex=[sp.exp(v) for v in z]\ns=sum(ex)\np=[e/s for e in ex]\nL=-sp.log(p[0])\ng0=p[0]-1\nresult = abs(float(p[0])-0.659)<0.002 and abs(float(L)-0.417)<0.002 and abs(float(g0)+0.341)<0.002"},
  {"claim":"Q3b He std for n=256 is 0.0884","code":"import sympy as sp\nvar=sp.Rational(2,256)\nstd=sp.sqrt(var)\nresult = abs(float(std)-0.08839)<0.001"},
  {"claim":"Q5 gradients df/dx=14.437, df/dy=5.683","code":"import sympy as sp\nx,y=sp.symbols('x y')\nf=(x*y+sp.sin(x))**2\nfx=sp.diff(f,x).subs({x:1.0,y:2.0})\nfy=sp.diff(f,y).subs({x:1.0,y:2.0})\nresult = abs(float(fx)-14.437)<0.01 and abs(float(fy)-5.683)<0.01"}
]