Level 4 — ApplicationResearch Frontiers & Practice

Research Frontiers & Practice

⏱ 60 minutes60 marksprintable — key stays hidden on paper

Level: 4 — Application (novel/unseen problems, no hints) Time limit: 60 minutes Total marks: 60

Question 1 — Contrastive Learning Objective (14 marks)

You are reproducing a SimCLR-style pipeline. For a batch of $N$ images, each image is augmented twice to give $2N$ views. The normalized-temperature cross-entropy (NT-Xent) loss for a positive pair $(i,j)$ is:

$\ell_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j)/\tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]}\exp(\text{sim}(z_i, z_k)/\tau)}$

where $\text{sim}(u,v) = \frac{u^\top v}{\lVert u\rVert \lVert v\rVert}$ is cosine similarity.

(a) Consider $N=2$ (so $2N=4$ embeddings). All embeddings are unit-normalized. The cosine similarities to view $i=1$ (whose positive is $j=2$ ) are: $\text{sim}(z_1,z_2)=0.8$ , $\text{sim}(z_1,z_3)=0.1$ , $\text{sim}(z_1,z_4)=-0.2$ . Using $\tau = 0.5$ , compute $\ell_{1,2}$ numerically to 3 decimal places. (6 marks)

(b) Explain what happens to $\ell_{1,2}$ in the limit $\tau \to 0^+$ when the positive similarity is strictly the largest, versus when a negative has higher similarity than the positive. (4 marks)

(c) A colleague removes L2-normalization from the embeddings but keeps the same loss form. Give two concrete failure modes this introduces and how each corrupts the training signal. (4 marks)

Question 2 — Federated Averaging (12 marks)

A federated system has 3 clients with local dataset sizes $n_1 = 100$ , $n_2 = 300$ , $n_3 = 600$ . After one local epoch each returns a scalar model parameter: $w_1 = 2.0$ , $w_2 = 5.0$ , $w_3 = 3.0$ .

(a) Compute the FedAvg aggregated global parameter $w_g = \sum_k \frac{n_k}{n} w_k$ . (4 marks)

(b) Client 3's data is drawn from a distribution very different from clients 1 and 2 (non-IID). Explain, using your numeric result, why weighting purely by sample count can be problematic here, and propose one alternative aggregation strategy. (4 marks)

(c) A privacy auditor claims "FedAvg is private because raw data never leaves the device." Critically evaluate this claim, naming one attack vector that survives and one mitigation. (4 marks)

Question 3 — Benchmark Design & Evaluation Rigor (12 marks)

A paper reports that Model A beats Model B by 0.4% top-1 accuracy on a benchmark, using a single train/test run each.

(a) Give three distinct methodological flaws that undermine the "beats" claim, and state the fix for each. (6 marks)

(b) You are told: 5 seeds give Model A a mean accuracy of $76.2\%$ with sample standard deviation $0.5\%$ , and Model B a mean of $75.8\%$ with sample standard deviation $0.6\%$ . Using a two-sample t-test intuition (equal-size groups), compute the pooled standard error of the difference and comment on whether the $0.4\%$ gap looks statistically convincing. (4 marks)

(c) Define data leakage in the context of benchmark design and give one non-obvious example specific to self-supervised pretraining. (2 marks)

Question 4 — Neuro-Symbolic & World Models (12 marks)

(a) You must build an agent that plays a grid puzzle where valid moves obey hard logical constraints (a symbolic rulebook) but perception of the grid comes from raw pixels. Design a neuro-symbolic architecture: name the neural component, the symbolic component, and the interface between them. State one advantage over a pure end-to-end deep RL agent. (6 marks)

(b) A world model predicts the next latent state $\hat{s}_{t+1} = f(s_t, a_t)$ and is used for planning by rolling out $H$ steps. If per-step latent prediction error is bounded by $\epsilon$ and the dynamics function $f$ is $L$ -Lipschitz in $s$ , argue how planning error can grow with horizon $H$ and give the resulting bound form. (4 marks)

(c) State one reason model-based planning with a learned world model can outperform model-free RL in sample efficiency. (2 marks)

Question 5 — Reproducibility, Open Source & Roadmap (10 marks)

(a) You attempt to reproduce a paper's results but get 3% lower accuracy. List four specific, checkable sources of the discrepancy (excluding "the authors cheated"). (4 marks)

(b) You want to contribute a bug fix to a large open-source ML library. Put these in the correct workflow order and justify one ordering choice: open a PR, write a failing test that reproduces the bug, fork & branch, fix the code so the test passes, open/link an issue. (3 marks)

(c) Propose a 3-milestone research roadmap for a student aiming to publish a first workshop paper on continual learning within 6 months. Each milestone must have a concrete deliverable. (3 marks)

Answer keyMark scheme & solutions

Question 1

(a) [6 marks]

Numerator: $\exp(0.8/0.5) = \exp(1.6)$ . Denominator sums over $k \neq 1$ : $k=2,3,4$ : $\exp(0.8/0.5) + \exp(0.1/0.5) + \exp(-0.2/0.5) = \exp(1.6)+\exp(0.2)+\exp(-0.4)$ .

$\exp(1.6) = 4.9530$
$\exp(0.2) = 1.2214$
$\exp(-0.4) = 0.6703$

Denominator $= 4.9530 + 1.2214 + 0.6703 = 6.8447$ . [2 marks for correct terms]

$\ell_{1,2} = -\log(4.9530/6.8447) = -\log(0.72363) = 0.3234$ . [2 marks arithmetic]

Answer: $\ell_{1,2} \approx 0.323$ . [2 marks final]

(b) [4 marks] As $\tau \to 0^+$ , the softmax sharpens toward a one-hot over similarities (2 marks).

If the positive similarity is strictly largest, its softmax probability $\to 1$ , so $\ell_{1,2} \to 0$ (1 mark).
If some negative has higher similarity, the positive probability $\to 0$ and $\ell_{1,2} \to \infty$ — the loss becomes dominated by the hardest (highest-sim) negative, i.e., very hard-negative focused (1 mark).

(c) [4 marks] Any two of:

Embedding norms become free parameters; the model can trivially minimize/inflate logits via magnitude rather than direction, decoupling loss from semantic alignment (2).
Loss scale becomes coupled to norm, causing exploding/unstable logits and gradient blow-up (2).
Cosine geometry (bounded $[-1,1]$ ) is lost; temperature $\tau$ no longer has a calibrated effect, so the effective "hardness" of negatives is uncontrolled (2).

Question 2

(a) [4 marks] $n = 100+300+600 = 1000$ . $w_g = \frac{100}{1000}(2.0) + \frac{300}{1000}(5.0) + \frac{600}{1000}(3.0)$ $= 0.2 + 1.5 + 1.8 = 3.5$ . [answer 3.5]

(b) [4 marks] Client 3 dominates the average (weight 0.6) and pulls $w_g$ toward its value even though its distribution differs; the aggregate is skewed toward the majority-count client regardless of representativeness (2). Alternative: cluster/personalized FedAvg, robust aggregation (median/trimmed mean, Krum), or reweighting by validation performance rather than sample count (2).

(c) [4 marks] Claim is only partially true — raw data stays local but gradients/updates leak information (2). Attack vector: gradient inversion / membership inference can reconstruct or infer training samples from updates (1). Mitigation: differential privacy (noise + clipping) or secure aggregation so the server sees only the sum (1).

Question 3

(a) [6 marks — 2 each, any three]

Single run → no variance estimate. Fix: multiple seeds + report mean±std/CI.
No significance test for 0.4% gap. Fix: paired statistical test across seeds.
Possible hyperparameter tuning asymmetry / tuned on test set. Fix: equal tuning budget, separate validation set.
Same-architecture confound / different compute budgets. Fix: matched FLOPs/parameters.

(b) [4 marks] Standard error of each mean: $SE_i = s_i/\sqrt{n}$ , $n=5$ . Pooled SE of difference: $SE_{\text{diff}} = \sqrt{\frac{s_A^2 + s_B^2}{n}} = \sqrt{\frac{0.25 + 0.36}{5}} = \sqrt{0.122} = 0.3493\%$ . [2 marks] $t = 0.4/0.3493 \approx 1.145$ . This is well below ~2.3 (critical for ~8 dof at 95%), so the gap is not statistically convincing — likely within noise. [2 marks]

(c) [2 marks] Data leakage = information from the test set influencing training/model selection, inflating reported performance (1). SSL-specific example: pretraining on a large corpus that already contains (or overlaps with) the downstream benchmark's test images, so "self-supervised" features encode test data (1).

Question 4

(a) [6 marks]

Neural component: CNN/perception network mapping pixels → symbolic grid state (object/cell predicates) (2).
Symbolic component: logic engine / constraint solver / rule checker that enumerates or validates legal moves (2).
Interface: neural net outputs discrete symbols/probabilities that are grounded into the symbolic KB; symbolic reasoner filters/plans over them (differentiable relaxation or hard grounding) (1).
Advantage: guaranteed constraint satisfaction, better generalization to unseen configurations, and data efficiency vs pure deep RL which must learn rules implicitly (1).

(b) [4 marks] Errors compound through the rollout: at each step both the injected per-step error $\epsilon$ and Lipschitz amplification of prior error accumulate. Total error after $H$ steps $\lesssim \epsilon \sum_{t=0}^{H-1} L^t = \epsilon \frac{L^H - 1}{L-1}$ (for $L\neq1$ ) (3). For $L>1$ this grows exponentially in $H$ ; for $L=1$ it grows linearly ( $\epsilon H$ ) (1).

(c) [2 marks] The learned model lets the agent generate synthetic rollouts / plan without new environment interactions, reusing each real sample many times → higher sample efficiency.

Question 5

(a) [4 marks — 1 each] Framework/library version differences; different random seeds/nondeterministic ops (cuDNN); hyperparameter/preprocessing mismatch (LR schedule, augmentation, normalization); hardware/precision (fp16 vs fp32), batch size, or undocumented tricks; different data split/version.

(b) [3 marks] Order: (1) open/link an issue → (2) fork & branch → (3) write a failing test that reproduces the bug → (4) fix the code so the test passes → (5) open a PR (2 marks). Justification: writing the failing test before the fix proves the bug exists and prevents regression / confirms the fix truly resolves it (1 mark).

(c) [3 marks — 1 each]

M1 (Month 1–2): reproduce a baseline continual-learning method (e.g., EWC) on a standard benchmark → deliverable: reproduced results + code repo.
M2 (Month 3–4): implement a novel tweak/hypothesis → deliverable: ablation results vs baseline with significance testing.
M3 (Month 5–6): write and submit → deliverable: 4-page workshop paper + reproducible artifact.

[
  {"claim":"NT-Xent loss l_1,2 ~ 0.323","code":"import math\nnum=math.exp(1.6)\nden=math.exp(1.6)+math.exp(0.2)+math.exp(-0.4)\nl=-math.log(num/den)\nresult=abs(l-0.323)<0.001"},
  {"claim":"FedAvg aggregate = 3.5","code":"wg=(100*2.0+300*5.0+600*3.0)/1000\nresult=abs(wg-3.5)<1e-9"},
  {"claim":"Pooled SE of difference ~0.3493 and t~1.145","code":"import math\nse=math.sqrt((0.25+0.36)/5)\nt=0.4/se\nresult=abs(se-0.3493)<0.001 and abs(t-1.145)<0.01"},
  {"claim":"Lipschitz error sum L=1 gives linear H*eps: for eps=0.1,H=5 -> 0.5","code":"eps=0.1;H=5;L=1\nerr=eps*H if L==1 else eps*(L**H-1)/(L-1)\nresult=abs(err-0.5)<1e-9"}
]