Research Frontiers & Practice

Question 1 (12 marks)

(a) (3 marks)

• The loss is a softmax over similarities where $z_i$ is the anchor. Including $k=i$ would put $\text{sim}(z_i,z_i)=1$ (self-similarity, trivially maximal) in the denominator, which is meaningless as a "negative" and would dominate — hence excluded. (2)
• $k=j$ is kept because $z_j$ is the positive whose similarity forms the numerator; it must also appear in the denominator so the expression is a proper normalized probability (numerator ⊆ denominator terms). (1)

(b) (6 marks) Similarities: $s = \{1.0, 0.8, 0.1, 0.0\}$, positive $j=2 \Rightarrow s_{ij}=0.8$. Exclude $k=i=1$. Scale by $1/\tau = 2$:

• numerator: $\exp(0.8/0.5)=\exp(1.6)=4.9530$ (2)
• denominator (exclude $k=1$): $\exp(1.6)+\exp(0.2)+\exp(0.0) = 4.9530 + 1.2214 + 1.0000 = 7.1744$ (2)
• $\ell = -\log(4.9530/7.1744) = -\log(0.69051) = 0.3705$ (2)

Answer: $\ell_{1,2} \approx 0.370$

(c) (3 marks) Sum over all $2N$ views, each acting as anchor with its own positive, and average: $\mathcal{L} = \frac{1}{2N}\sum_{k=1}^{N}\big[\ell_{2k-1,2k} + \ell_{2k,2k-1}\big]$ Both directions of each positive pair are counted; loss is symmetric per pair but computed with each view as anchor. (3)

Question 2 (10 marks)

(a) (5 marks)

import numpy as np
def layernorm(x, gamma, beta, eps=1e-5):
    # x shape (..., H), normalize over last axis
    mu = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    xhat = (x - mu) / np.sqrt(var + eps)
    return gamma * xhat + beta

Marks: mean (1), var over last axis (1), normalize (1), affine $\gamma,\beta$ (1), eps inside sqrt (1).

(b) (5 marks) With $\hat{x}_i = (x_i-\mu)/\sigma$, $\sigma=\sqrt{\sigma^2}$:

• $\frac{\partial \mu}{\partial x_j} = \frac{1}{H}$ (1)
• $\frac{\partial \sigma^2}{\partial x_j} = \frac{2}{H}(x_j-\mu) = \frac{2}{H}\sigma\hat{x}_j$, so $\frac{\partial \sigma}{\partial x_j} = \frac{1}{H}\hat{x}_j$ (1)
• Quotient rule: $\frac{\partial \hat{x}_i}{\partial x_j} = \frac{(\delta_{ij}-\tfrac1H)\sigma - (x_i-\mu)\tfrac1H\hat{x}_j}{\sigma^2}$ (1)
• Substitute $x_i-\mu = \sigma\hat{x}_i$: $= \frac{\delta_{ij}-\tfrac1H}{\sigma} - \frac{\hat{x}_i\hat{x}_j}{H\sigma} = \frac{1}{\sigma}\left(\delta_{ij} - \frac{1}{H} - \frac{\hat{x}_i\hat{x}_j}{H}\right)$ (2)

Question 3 (10 marks)

(a) (4 marks, 1 each)

1. Tuning on the test set / no held-out validation — hyperparameters chosen using test data inflate results.
2. Unfair baselines — baselines under-tuned or run with fewer epochs/compute than the proposed method.
3. No variance reporting / single seed — no error bars; gap could be within run-to-run noise.
4. Data leakage / contamination — train-test overlap, or pretraining data covering the benchmark. (Also accept: cherry-picked metrics, non-standard splits, missing ablations.)

(b) (6 marks)

• Standard error of a proportion: $SE = \sqrt{p(1-p)/n}$ with $p=0.91$, $n=2000$. (1)
• $p(1-p)=0.91\times0.09=0.0819$; $/2000 = 4.095\times10^{-5}$; $\sqrt{} = 0.006399$. (2)
• 95% half-width $= 1.96 \times SE = 1.96 \times 0.006399 = 0.01254 \approx 1.25\%$. (2)
• The observed gap is $0.4\%$, which is smaller than the ~1.25% CI half-width of a single system (and a difference test on two proportions would give a comparable-or-larger threshold). The 0.4% gap is NOT likely statistically significant with only 2000 samples/one seed. (1)

Question 4 (10 marks)

(a) (3 marks) $w^{t+1} = \sum_{k=1}^{K} \frac{n_k}{n}\, w_k^{t+1}$ Weighted average by local dataset size. (3)

(b) (5 marks) Assumption: all clients start the round from the same global weights $w^t$ and each does one full-batch step with learning rate $\eta$. (1) Local update: $w_k^{t+1} = w^t - \eta g_k$ where $g_k = \nabla F_k(w^t) = \frac{1}{n_k}\sum_{x\in D_k}\nabla \ell(w^t,x)$. (1) Aggregate: $w^{t+1} = \sum_k \frac{n_k}{n}(w^t - \eta g_k) = w^t - \eta \sum_k \frac{n_k}{n} g_k$ (1) The weighted gradient equals the global gradient: $\sum_k \frac{n_k}{n} g_k = \sum_k \frac{n_k}{n}\cdot\frac{1}{n_k}\sum_{x\in D_k}\nabla\ell = \frac{1}{n}\sum_{x\in D}\nabla\ell = \nabla F(w^t)$ (1) Thus $w^{t+1} = w^t - \eta\nabla F(w^t)$ = one centralized full-batch GD step. (1)

(c) (2 marks) With multiple local epochs, clients diverge from $w^t$ (client drift): their gradients are evaluated at different points $w_k \neq w^t$, so the average of local updates no longer equals a single gradient at $w^t$ — the equivalence breaks (worse under non-IID data).

Question 5 (10 marks)

(a) (5 marks)

• Purely neural: everything is learned distributed representations; reasoning is implicit in weights, hard to interpret, needs lots of data. (2)
• Neuro-symbolic: combines learned perception with explicit symbolic structure (logic rules, programs, knowledge graphs) that support compositional, verifiable reasoning and data efficiency. (2)
• Concrete example: e.g. Neural Theorem Provers, DeepProbLog, or a VQA system where a CNN extracts objects and a differentiable program executor answers logical queries. (1)

(b) (5 marks)

• Encoder: maps high-dim observations (pixels) to compact latent state $z$. (1)
• Latent dynamics/transition: learns $z_{t+1} = f(z_t, a_t)$ — predicting how state evolves given actions. (1)
• Decoder/reward head: reconstructs observations and/or predicts reward, providing training signal / grounding. (1)
• Latent-space dynamics preferred because: pixel prediction wastes capacity on irrelevant detail; latent rollouts are far cheaper/faster for planning; and imagining trajectories in a low-dim, smooth space is more stable and generalizes better (as in Dreamer). (2)

Question 6 (8 marks)

(a) (3 marks, 1 each)

1. Verify exact hyperparameters & schedule (LR, warmup, batch size, epochs, weight decay) — often under-reported; try author configs.
2. Match data preprocessing/augmentation and splits precisely, plus dataset version.
3. Account for compute/precision/seed variance — run multiple seeds, check hardware (fp16 vs fp32), effective batch size differences from distributed training. (Also accept: contact authors, check released code/checkpoints, EMA of weights.)

(b) (5 marks) Example roadmap:

• Month 1–2: reproduce 2–3 core papers end-to-end; publish clean repos + blog write-ups (milestone: matched numbers). (1)
• Month 3: implement a key model from scratch and ablate components to build intuition. (1)
• Month 4: identify a gap/open problem from a survey; form a concrete hypothesis and design a benchmark/experiment. (1)
• Month 5: run experiments with rigor (baselines, seeds, error bars); iterate. (1)
• Month 6: write up findings (workshop paper / preprint), open-source code, present. Portfolio = repos + reproductions + one original result. (1)

[
  {"claim":"NT-Xent toy loss equals ~0.370",
   "code":"import sympy as sp\nnum=sp.exp(sp.Rational(8,10)/sp.Rational(1,2))\nden=num+sp.exp(sp.Rational(1,10)/sp.Rational(1,2))+sp.exp(0)\nl=-sp.log(num/den)\nval=float(l)\nresult=abs(val-0.370)<0.002"},
  {"claim":"Proportion SE at p=0.91,n=2000 gives 95% half-width ~0.0125",
   "code":"p=sp.Rational(91,100);n=2000\nse=sp.sqrt(p*(1-p)/n)\nhw=1.96*se\nresult=abs(float(hw)-0.01254)<0.0005"},
  {"claim":"FedAvg weighted gradient equals global gradient (symbolic identity)",
   "code":"n1,n2,g1,g2=sp.symbols('n1 n2 g1 g2')\nn=n1+n2\nweighted=(n1/n)*g1+(n2/n)*g2\nglobal_grad=(n1*g1+n2*g2)/n\nresult=sp.simplify(weighted-global_grad)==0"},
  {"claim":"LayerNorm Jacobian formula reduces correctly for i=j term",
   "code":"H,sigma,xi,xj=sp.symbols('H sigma xi xj')\nexpr=(1/sigma)*(1-sp.Rational(1,1)/H - xi*xj/H)\n# check off-diagonal (delta=0) form: -(1/sigma)*(1/H)*(1+xi*xj)\noffdiag=(1/sigma)*(0-1/H-xi*xj/H)\nexpected=-(1/(sigma*H))*(1+xi*xj)\nresult=sp.simplify(offdiag-expected)==0"}
]