Research Frontiers & Practice

Question 1

(a) InfoNCE mutual information bound (7)

Setup (1): Treat the positive as a classification-over-$K$ problem: the numerator is the true (positive) density, the denominator sums $K$ terms drawn as 1 positive + $(K-1)$ negatives.

Derivation (4): The optimal InfoNCE critic satisfies $f(z_i,z_j)\propto \frac{p(z_i|z_j)}{p(z_i)}$. Substituting the optimal critic: $\mathbb{E}[\ell] = -\mathbb{E}\!\left[\log \frac{p(z_i|z_j)/p(z_i)}{p(z_i|z_j)/p(z_i)+\sum_{neg}p(z_i|z_j')/p(z_i)}\right].$ Bounding the negative sum by its expectation ($\approx (K-1)$) and using Jensen: $\mathbb{E}[\ell] \geq \log K - I(z_i;z_j).$ Hence $I(z_i;z_j) \geq \log K - \mathbb{E}[\ell]$: minimizing $\ell$ raises the MI lower bound. (2 for final inequality direction)

Assumptions (2): (i) negatives are i.i.d. from the marginal $p(z)$; (ii) critic is expressive enough to reach the optimum; (iii) large-batch approximation for the expectation of the negative sum. The bound is tighter as $K$ (batch size) grows — motivating SimCLR's large batches.

(b) Gradients & temperature (7)

Let $s_k=\mathrm{sim}(z_i,z_k)/\tau$ and softmax $p_k = \frac{\exp s_k}{\sum_{m\neq i}\exp s_m}$. Then: $\frac{\partial \ell_{i,j}}{\partial\, \mathrm{sim}(z_i,z_k)} = \frac{1}{\tau}\big(p_k - \mathbf{1}_{[k=j]}\big).$

• For a negative $k$: gradient $=\frac{1}{\tau}p_k>0$ → loss pushes similarity down, weighted by softmax probability. (3)
• For the positive $j$: gradient $=\frac{1}{\tau}(p_j-1)<0$ → pulls positive similarity up. (2)

Temperature interpretation (2): As $\tau\to 0^+$, $s_k$ blows up; softmax $p_k$ concentrates on the largest $s_k$ (the hardest negative, highest cosine sim). Thus the gradient mass focuses on the single hardest negative — sharp, hard-negative-mining behaviour. Large $\tau$ → uniform $p_k$ → all negatives weighted equally (soft). The $1/\tau$ prefactor amplifies gradient magnitude at small $\tau$.

(c) Vectorized NT-Xent pseudocode + complexity (8)

# Z: (2N, d), already L2-normalized
S = Z @ Z.T / tau          # (2N, 2N) similarity/temperature
S.fill_diagonal_(-inf)     # mask self-similarity (i != k)
# positive index for row i: its paired augmentation j
targets = pairing_indices  # length 2N, maps i -> j
loss = cross_entropy(S, targets)   # softmax over rows, pick target col
return loss.mean()

Marks: correct sim matrix (2), diagonal masking (2), positive targeting via cross-entropy (2).

Complexity (2): Matrix $ZZ^\top$: time $O((2N)^2 d)=O(N^2 d)$, memory $O(N^2)$ for the similarity matrix. Diagonal mask explanation: term $k=i$ (self-similarity $=1$ after norm) is excluded by the indicator $\mathbf 1_{[k\neq i]}$; leaving it in would place a trivial maximal logit on the self entry, corrupting the softmax normalization.

Question 2

(a) Client drift & effect of $E$ (8)

Convergence intuition (3): FedAvg bound (Li et al. 2020) has form $\mathbb{E}[F(w_T)-F^\star] \leq \frac{\kappa}{T+\gamma}\Big(\underbrace{B}_{\text{variance}} + \underbrace{C\Gamma}_{\text{heterogeneity}} + \dots\Big),$ where terms multiplying $E$ and $\Gamma$ grow with more local steps. Larger $E$ lets each client descend toward its own $F_m^\star$, which differs from $F^\star$ by heterogeneity $\Gamma$ → the averaged model sits between local optima, not at the global one ("client drift").

Drift derivation (5): After $E$ local steps from shared $w^t$, client $m$'s weight is $w_m^{t,E} = w^t - \eta\sum_{e=0}^{E-1}\nabla F_m(w_m^{t,e}).$ To first order (freeze gradient at $w^t$): $w_m^{t,E} \approx w^t - \eta E\,\nabla F_m(w^t)$. Drift relative to a global SGD step $w^t-\eta E\nabla F(w^t)$: $\delta_m = w_m^{t,E} - \big(w^t-\eta E \nabla F(w^t)\big) = -\eta E\big(\nabla F_m(w^t) - \nabla F(w^t)\big).$ Its magnitude scales with $\eta E \cdot \|\nabla F_m - \nabla F\|$ — proportional to $E$ and to gradient dissimilarity (heterogeneity). Hence large $E$ + high $\Gamma$ → large drift → biased average.

(b) DP noise computation (7)

$\sigma \geq \frac{C\sqrt{2\ln(1.25/\delta)}}{\varepsilon} = \frac{1.0\cdot\sqrt{2\ln(125000)}}{1.0}.$ $\ln(1.25/10^{-5}) = \ln(125000) \approx 11.7361$. $2\times 11.7361 = 23.472$; $\sqrt{23.472}\approx 4.845$. $\boxed{\sigma \approx 4.85}$ (4 for value + method)

Effect (3): Added noise variance $\propto \sigma^2 C^2$ enters the convergence variance term additively (scaled by $1/M$ if noise averages across $M$ clients: variance term $\sim \sigma^2 C^2 d / M$). Tradeoff: smaller $\varepsilon$ (stronger privacy) → larger $\sigma$ → larger variance floor → higher final loss / slower convergence. Averaging over many clients mitigates it ($1/M$).

(c) Neuro-symbolic federated (5)

Advantage (2): Symbolic rules (e.g., logical predicates / decision rules) are tiny compared to full gradient/weight tensors → drastically lower communication bandwidth and no per-round dense transmission. Open problem (2): Rule conflict resolution — clients may learn contradictory rules; merging requires a sound aggregation logic (truth maintenance, weighted voting, consistency checking) with no established, differentiable-friendly solution. (+1 for coherent framing.)

Question 3

(a) Confidence interval (6)

Wald 95% half-width: $z\sqrt{\hat p(1-\hat p)/n}$ with $z=1.96$. $\hat p(1-\hat p)=0.751\times0.249=0.186999$. Divide by $n$: $0.186999/50000=3.740\times10^{-6}$. Sqrt $=1.9339\times10^{-3}$. Times $1.96$: $\text{half-width} \approx 0.00379 \approx 0.38\%.$ (4)

Judgement (2): 95% CI $\approx 75.1\%\pm0.38\%$. A $+2.1\%$ gap ($\approx 5.5\times$ the half-width) exceeds the sampling noise on this test set, so it is statistically distinguishable from test-set sampling noise alone. BUT with a single run, the claim ignores training-seed/optimization variance, which is typically larger than test-sampling noise — so the claim is not established without multiple runs.

(b) Reproduction controls (7) — 5 × ~1.4 marks

1. Multiple seeds + report mean±std — prevents mistaking a lucky run for real improvement (training variance).
2. Compute/FLOP-matched baseline — prevents attributing gains to more compute/params rather than the method.
3. Pretraining–eval data leakage audit — prevents inflated zero-shot scores from test images in the pretrain corpus.
4. Hyperparameter search parity — equal tuning budget for baseline and new method; prevents unfair under-tuned baselines.
5. Fixed data splits + preprocessing release — prevents evaluation-protocol drift and cherry-picked splits. (Accept: released code/checkpoints, ablations isolating the claimed component.)

(c) Deduplication + leakage audit (5)

Method (3): Embed all pretraining images and all ImageNet test images with a frozen, independent perceptual hasher or embedding model; compute nearest-neighbour cosine similarity; flag pairs above a threshold (or exact/near-dup via perceptual hashing like pHash). Also match on caption text for CLIP-style duplicates. Report the % of test images with a near-duplicate in pretraining and re-evaluate on the deduplicated subset. Why leakage inflates (2): If test images (or their captions) appear in pretraining, the model has effectively memorized the label association, so "zero-shot" accuracy measures memorization, not generalization — biasing the metric upward and voiding the SOTA comparison.

[
{"claim":"InfoNCE negative-logit gradient prefactor is 1/tau, positive is (p-1)/tau; sign of negative gradient positive","code":"tau=Rational(1,2); s0,s1,s2=symbols('s0 s1 s2'); den=exp(s0/tau)+exp(s1/tau)+exp(s2/tau); ell=-log(exp(s0/tau)/den); g_neg=diff(ell,s1); val=g_neg.subs({s0:0,s1:0,s2:0}); result=(simplify(val)>0)==True","comment":"positive gradient on a negative logit"},
{"claim":"DP sigma for eps=1, delta=1e-5, C=1 is approx 4.845","code":"sig=1.0*sqrt(2*ln(1.25/Rational(1,100000)))/1.0; result=abs(float(sig)-4.845)<0.01"},
{"claim":"Wald 95% half-width for p=0.751,n=50000 is approx 0.00379","code":"p=Rational(751,1000); n=50000; hw=1.96*sqrt(p*(1-p)/n); result=abs(float(hw)-0.00379)<0.0001"},
{"claim":"2.1% gap exceeds Wald half-width (distinguishable from sampling noise)","code":"p=Rational(751,1000); n=50000; hw=float(1.96*sqrt(p*(1-p)/n)); result=(0.021>hw)==True"}
]