AI Safety & Alignment

Q1 (12)

(a) $M$ is $(\varepsilon,\delta)$-DP if for all neighbouring $D,D'$ (differing in one record) and all measurable $S\subseteq\text{Range}(M)$: $\Pr[M(D)\in S]\le e^{\varepsilon}\Pr[M(D')\in S]+\delta.$ Marks: definition 2, neighbouring/all-$S$ quantifiers 1.

(b) Laplace density $p(x)=\frac{1}{2b}e^{-|x|/b}$. For output $z$, ratio of densities: $\frac{p_{M(D)}(z)}{p_{M(D')}(z)}=\frac{\exp(-|z-f(D)|/b)}{\exp(-|z-f(D')|/b)}=\exp\!\Big(\frac{|z-f(D')|-|z-f(D)|}{b}\Big).$ By reverse triangle inequality $|z-f(D')|-|z-f(D)|\le |f(D)-f(D')|\le \Delta_1 f$. So ratio $\le e^{\Delta_1 f/b}$. To enforce $\le e^{\varepsilon}$ we need $\Delta_1 f/b\le\varepsilon$, i.e. $b=\frac{\Delta_1 f}{\varepsilon}.$ Marks: density 1, ratio 2, triangle-inequality bound 2, conclusion 1.

(c) $b=\Delta_1 f/\varepsilon = 1/0.5 = 2$. Variance $=2b^2 = 2(4)=8$. Marks: $b$ 1.5, variance 1.5.

Q2 (11)

(a) Demographic parity: $P(\hat Y=1\mid G=A)=P(\hat Y=1\mid G=B)$. Equal opportunity: $P(\hat Y=1\mid Y=1,G=A)=P(\hat Y=1\mid Y=1,G=B)$ (equal TPR). Marks 1.5 each.

(b) Predicted positive $=TP+FP$; total $=TP+FP+FN+TN$.

• A: $(30+10)/100 = 0.40$.
• B: $(45+30)/100 = 0.75$. DP difference $=|0.40-0.75| = 0.35$. Marks: each rate 1.5, difference 1.

(c) TPR $=TP/(TP+FN)$.

• A: $30/50 = 0.60$.
• B: $45/50 = 0.90$. Difference $=0.30 > 0.05$ → equal opportunity does not hold. Marks: each TPR 1.5, verdict 1.

Q3 (9)

(a) Reward hacking: the agent optimizes the specified reward but achieves high reward through unintended behaviour that doesn't fulfil the designer's intent (the reward is a flawed proxy). Example: boat-racing agent looping to collect points instead of finishing the race. Goal misgeneralization: capabilities generalize but the learned goal is a wrong proxy that coincided with the true goal in training; the model competently pursues the wrong goal off-distribution even with a correct reward. Example: agent trained to reach a coin at level-end learns "go right" and ignores relocated coins. Marks: 2 definitions + 2 examples.

(b) Gaming ways (2×2): (1) obscure dirt from its sensors (cover with a mat, turn off camera) so "visible dirt" reads zero; (2) push dirt out of view / into corners rather than removing it. Fix (2): base reward on ground-truth cleanliness measured by an independent held-out sensor / redefine reward over actual dirt mass removed rather than perceived visible dirt, plus penalize sensor tampering. Marks: 2 valid exploits (2), plausibility (1), reasonable fix (2).

Q4 (9)

(a) Outer alignment: the specified objective/reward (the training signal) correctly captures the designer's true intent — aligning the objective with human values. Inner alignment: the goal the trained model actually internalizes (the mesa-objective) matches the specified training objective — aligning the learned model with the objective. Marks: 2 + 2.

(b) This is an inner-alignment failure, specifically goal misgeneralization. The objective (reach green door) was fine (outer alignment ok), but the model learned a spurious proxy correlated in training ("go left"/go to the salient object on the left). Competent action toward the wrong goal off-distribution = inner misalignment. Marks: correct class (2), sub-type (1), justification distinguishing proxy from bad reward (2).

Q5 (10)

(a) Adversary has query access to a target model and a candidate example; goal is to infer whether that example was in the training set. Motivation: privacy leakage; models often assign higher confidence / lower loss to training members. Marks 3.

(b)

# Calibration using a shadow model with known membership
shadow = train_model(shadow_train)
conf_in  = [max_softmax(shadow(x)) for x in shadow_train]     # members
conf_out = [max_softmax(shadow(x)) for x in shadow_holdout]   # non-members
tau = choose_threshold(conf_in, conf_out)  # e.g. maximize accuracy / ROC point

def MIA(target_model, x):
    c = max_softmax(target_model(x))   # or use loss: lower loss -> member
    return "member" if c >= tau else "non-member"

Marks: confidence signal 1.5, threshold decision 1.5, shadow calibration 2.

(c) Defenses: DP-SGD (bounds per-example influence so member/non-member outputs are $e^\varepsilon$-indistinguishable), or regularization/confidence-masking to shrink the train-vs-test confidence gap. Marks 2.

Q6 (9)

(a) Four tiers: Unacceptable risk (banned — e.g. social scoring, real-time public biometric ID); High risk (e.g. medical devices, hiring/CV screening, credit scoring — strict conformity assessment); Limited risk (transparency obligations — e.g. chatbots, deepfakes must be labelled); Minimal risk (e.g. spam filters, AI in video games — no obligations). Marks 1 per tier+example.

(b) Catastrophic risk: massive harm to many people/society but recoverable. Existential risk: permanent curtailment of humanity's potential (extinction or irreversible lock-in). X-not-catastrophic example: an unaligned superintelligence permanently disempowering humanity / irreversible value lock-in — no recovery possible. Marks: distinction 2, scenario 1.

(c) Any two: staged/gated release with capability evals & red-teaming, model cards + usage policy, monitoring & incident response, RLHF safety fine-tuning, watermarking outputs. Marks 1 each.

[
 {"claim":"Laplace scale b=Delta/eps=2 for Delta=1,eps=0.5","code":"Delta=1; eps=Rational(1,2); b=Delta/eps; result=(b==2)"},
 {"claim":"Laplace noise variance 2b^2 = 8","code":"b=2; var=2*b**2; result=(var==8)"},
 {"claim":"Demographic parity difference = 0.35","code":"pA=Rational(30+10,100); pB=Rational(45+30,100); result=(abs(pA-pB)==Rational(35,100))"},
 {"claim":"TPR gap A=0.6 B=0.9 exceeds tol 0.05","code":"tprA=Rational(30,50); tprB=Rational(45,50); result=(tprA==Rational(3,5) and tprB==Rational(9,10) and abs(tprA-tprB)>Rational(1,20))"}
]