Level 2 — RecallNeural Network Fundamentals

Neural Network Fundamentals

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Level: 2 (Recall / Standard Problems) Time Limit: 30 minutes Total Marks: 40

Q1. State the perceptron update rule for a misclassified example $(\mathbf{x}, y)$ with $y \in \{-1, +1\}$ , and briefly explain what limitation of the single perceptron was highlighted by the XOR problem. (4 marks)

Q2. A single neuron receives inputs $\mathbf{x} = [2, -1, 3]$ with weights $\mathbf{w} = [0.5, 1.0, -0.5]$ and bias $b = 1$ . Compute the pre-activation $z = \mathbf{w}\cdot\mathbf{x} + b$ , then the output using the sigmoid activation $\sigma(z) = \frac{1}{1+e^{-z}}$ . (4 marks)

Q3. Write the definitions of the sigmoid and tanh activation functions. State the algebraic relationship between them and give the output range of each. (4 marks)

Q4. Define the ReLU function and Leaky ReLU (with parameter $\alpha$ ). Give the derivative of each. State one advantage of Leaky ReLU over standard ReLU. (5 marks)

Q5. Given logits $\mathbf{z} = [1, 2, 3]$ , compute the softmax output vector $\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$ . Round each component to 3 decimal places. (5 marks)

Q6. For a single training example, the target is $y = 1$ and the predicted probability is $\hat{y} = 0.8$ . Compute the binary cross-entropy loss $L = -[y\ln\hat{y} + (1-y)\ln(1-\hat{y})]$ . Also state the MSE for the same values. (4 marks)

Q7. State the Universal Approximation Theorem for a feedforward network with a single hidden layer. What does it not guarantee? (4 marks)

Q8. Explain what the vanishing gradient problem is, name one activation function that commonly causes it, and give one technique used to mitigate it. (4 marks)

Q9. State the He initialization variance formula for a layer with $n_{in}$ input units, and explain why He (rather than Xavier) is preferred for ReLU networks. (3 marks)

Q10. In one or two sentences, explain the role of the bias term in a neuron. What geometric effect does removing all biases have on a decision boundary $\mathbf{w}\cdot\mathbf{x}+b=0$ ? (3 marks)

Answer keyMark scheme & solutions

Q1. (4 marks) Update rule: $\mathbf{w} \leftarrow \mathbf{w} + \eta\, y\, \mathbf{x}$ , and $b \leftarrow b + \eta\, y$ , applied when $y(\mathbf{w}\cdot\mathbf{x}+b) \le 0$ (misclassified). (2 marks) XOR limitation: the single perceptron can only represent linearly separable functions; XOR is not linearly separable, so no single perceptron can solve it (Minsky & Papert, 1969). (2 marks)

Q2. (4 marks) $z = (0.5)(2) + (1.0)(-1) + (-0.5)(3) + 1 = 1 - 1 - 1.5 + 1 = -0.5$ . (2 marks) $\sigma(-0.5) = \frac{1}{1+e^{0.5}} = \frac{1}{1+1.6487} = 0.3775$ . (2 marks)

Q3. (4 marks) $\sigma(z) = \dfrac{1}{1+e^{-z}}$ , range $(0,1)$ . (1) $\tanh(z) = \dfrac{e^z - e^{-z}}{e^z + e^{-z}}$ , range $(-1,1)$ . (1) Relationship: $\tanh(z) = 2\sigma(2z) - 1$ . (2)

Q4. (5 marks) ReLU: $f(z) = \max(0, z)$ ; derivative $= 1$ if $z>0$ , else $0$ . (2) Leaky ReLU: $f(z) = z$ if $z>0$ , else $\alpha z$ ; derivative $=1$ if $z>0$ , else $\alpha$ . (2) Advantage: Leaky ReLU avoids "dying ReLU" — neurons with negative inputs still pass a small gradient, so they can recover during training. (1)

Q5. (5 marks) $e^1=2.7183,\ e^2=7.3891,\ e^3=20.0855$ ; sum $=30.1929$ . (2) softmax $= [2.7183/30.1929,\ 7.3891/30.1929,\ 20.0855/30.1929]$ $= [0.090,\ 0.245,\ 0.665]$ . (3)

Q6. (4 marks) BCE: $L = -[1\cdot\ln 0.8 + 0] = -\ln 0.8 = 0.2231$ . (2) MSE: $(y-\hat{y})^2 = (1-0.8)^2 = 0.04$ . (2)

Q7. (4 marks) Statement: A feedforward network with a single hidden layer containing a finite number of neurons and a suitable (non-polynomial/squashing) activation can approximate any continuous function on a compact subset of $\mathbb{R}^n$ to arbitrary accuracy. (2) It does not guarantee: (a) that the required number of neurons is small/practical, nor (b) that such weights are learnable/findable by training (no guarantee on optimization or generalization). (2)

Q8. (4 marks) Vanishing gradient: during backprop, gradients shrink multiplicatively across layers, becoming near-zero in early layers so their weights barely update. (2) Cause: sigmoid (or tanh), whose derivative is $\le 0.25$ (saturates). (1) Mitigation: use ReLU activations / proper (He) initialization / residual connections / batch normalization (any one). (1)

Q9. (3 marks) He: $\text{Var}(W) = \dfrac{2}{n_{in}}$ (weights drawn from $\mathcal{N}(0, 2/n_{in})$ ). (2) Preferred for ReLU because ReLU zeroes out ~half the activations, halving the variance; the factor $2$ (vs Xavier's $1/n_{in}$ scaling) compensates for this to keep signal/gradient variance stable. (1)

Q10. (3 marks) The bias allows the activation threshold to shift, letting the neuron fire for inputs not passing through the origin; it adds a constant offset independent of inputs. (2) Removing all biases forces every decision boundary $\mathbf{w}\cdot\mathbf{x}=0$ to pass through the origin, reducing representational flexibility. (1)

[
  {"claim":"Q2: z = -0.5 and sigmoid(z) approx 0.3775","code":"z=Rational(1,2)*2+1*(-1)+Rational(-1,2)*3+1; sig=1/(1+exp(-z)); result=(z==Rational(-1,2)) and abs(float(sig)-0.3775)<0.001"},
  {"claim":"Q5: softmax([1,2,3]) approx [0.090,0.245,0.665]","code":"import math; zs=[1,2,3]; es=[math.exp(v) for v in zs]; s=sum(es); sm=[e/s for e in es]; result=all(abs(sm[i]-t)<0.001 for i,t in enumerate([0.090,0.245,0.665]))"},
  {"claim":"Q6: BCE = -ln(0.8) approx 0.2231 and MSE = 0.04","code":"bce=-log(Rational(8,10)); mse=(1-Rational(8,10))**2; result=abs(float(bce)-0.2231)<0.001 and mse==Rational(1,25)"},
  {"claim":"Q3: tanh(z) = 2*sigmoid(2z)-1 identity","code":"z=symbols('z'); lhs=tanh(z); rhs=2*(1/(1+exp(-2*z)))-1; result=simplify(lhs-rhs)==0"}
]