Level 5 — MasteryAI Safety & Alignment

AI Safety & Alignment

⏱ 2 minutes100 marksprintable — key stays hidden on paper

Time limit: 2 hours 30 minutes Total marks: 100 Instructions: Answer ALL three questions. Show all derivations. Code may be written in Python-like pseudocode but must be precise. Calculators/CAS permitted for arithmetic checks only.

Question 1 — Differential Privacy, Fairness, and Their Tension (34 marks)

A hospital trains a binary classifier $f_\theta$ predicting disease. It releases model outputs under $(\varepsilon,\delta)$ -differential privacy via the Gaussian mechanism, and is simultaneously audited for group fairness across a protected attribute $A\in\{0,1\}$ .

(a) State the formal definition of $(\varepsilon,\delta)$ -differential privacy for a randomized mechanism $M$ over neighbouring datasets $D,D'$ . (4)

(b) For a query $q:\mathcal D\to\mathbb R^d$ with $\ell_2$ -sensitivity $\Delta_2 = \max_{D\sim D'}\lVert q(D)-q(D')\rVert_2,$ the Gaussian mechanism adds noise $\mathcal N(0,\sigma^2 I_d)$ . Prove that choosing $\sigma \ge \frac{\Delta_2\sqrt{2\ln(1.25/\delta)}}{\varepsilon}$ guarantees $(\varepsilon,\delta)$ -DP for the single release. You may use the privacy-loss random variable and the Gaussian tail bound $\Pr[Z>t]\le \tfrac12 e^{-t^2/2}$ . (10)

(c) Numerical: a gradient has clipping norm $C=1.0$ , so $\Delta_2 = C$ . You want $(\varepsilon,\delta)=(1.0,\,10^{-5})$ for a single step. Compute the minimum $\sigma$ from the bound in (b). (4)

(d) Composition: you run $T=1000$ noisy-SGD steps, each satisfying $(\varepsilon_0,\delta_0)$ -DP. Using basic (linear) composition, express the total $(\varepsilon_{\text{tot}},\delta_{\text{tot}})$ . Then explain in two sentences why advanced/moments-accountant composition gives a bound scaling like $O(\sqrt{T})$ in $\varepsilon$ rather than $O(T)$ . (6)

(e) Fairness metric. Let $\hat Y=f_\theta(X)$ . Define demographic parity gap and equalized-odds gap. Prove the impossibility statement: if base rates differ, $\Pr[Y=1\mid A=0]\ne\Pr[Y=1\mid A=1]$ , then a classifier with perfect calibration within groups and non-trivial accuracy cannot simultaneously satisfy demographic parity — sketch the argument formally. (10)

Question 2 — Reward Hacking, Inner Alignment, and Goal Misgeneralization (33 marks)

Consider an RL agent in an MDP $(\mathcal S,\mathcal A,P,R,\gamma)$ . The intended objective is a true reward $R^\star$ , but the designer specifies a proxy $\tilde R$ .

(a) Define outer alignment and inner alignment, and give one concrete failure example of each distinct from reward hacking. (6)

(b) Reward hacking as Goodhart. Suppose $\tilde R(s)=R^\star(s)+ \eta(s)$ where $\eta$ is a correlated "proxy error". Model the agent as choosing a state distribution $d$ to maximize $\mathbb E_{s\sim d}[\tilde R(s)]$ subject to $\mathbb E_{s\sim d}[c(s)]\le B$ (cost budget). Show, using a Lagrangian argument, that as the optimizer pushes $d$ toward the arg-max of $\tilde R$ , the correlation $\rho(R^\star,\tilde R)$ over the induced distribution can decrease even though it was high over the training distribution. Give the KKT stationarity condition and interpret it. (12)

(c) Goal misgeneralization: construct a minimal concrete example. A gridworld has a coin always placed at the top-right during training. Define two reward-consistent policies — "go to coin" vs "go to top-right" — that are indistinguishable on the training distribution but diverge under distribution shift when the coin is moved. Formalize why empirical risk minimization cannot prefer one over the other, and state one mitigation. (9)

(d) Give the precise distinction between reward hacking and goal misgeneralization in terms of whether the specified reward is optimized correctly. (6)

Question 3 — Adversarial Robustness, Jailbreaks, and Backdoors (33 marks)

(a) Define an $\ell_\infty$ adversarial example for a classifier $f$ with radius $\varepsilon$ , and write the untargeted FGSM perturbation. Prove that for a locally linear model $f(x)=w^\top x + b$ , the optimal $\ell_\infty$ -bounded perturbation of radius $\varepsilon$ that maximizes the logit change is $\delta^\star=\varepsilon\,\mathrm{sign}(w)$ , and compute the resulting change in output. (10)

(b) Certified robustness via Lipschitz bound. If $f$ has margin $m(x)=f_y(x)-\max_{j\ne y} f_j(x)$ and every logit is $L$ -Lipschitz in $\ell_2$ , prove no $\ell_2$ perturbation of norm $<\frac{m(x)}{2L}$ can change the prediction. (8)

(c) Backdoor / data poisoning. A trigger patch $t$ is added to a fraction $p$ of training images with flipped label. Model the poisoned expected loss as $(1-p)\mathbb E_{\text{clean}}[\ell] + p\,\mathbb E_{\text{trig}}[\ell]$ . Explain why a small $p$ (e.g. 1%) can achieve high attack success rate while barely moving clean accuracy, and name two defenses. (8)

(d) Jailbreaks. Define a jailbreak against an aligned LLM. Explain the mechanism by which "role-play" or "prefix-injection" attacks bypass a refusal-trained model in terms of the shift between the training distribution of harmful prompts and the adversarial distribution. Relate this conceptually to goal misgeneralization from Q2. (7)

Answer keyMark scheme & solutions

Question 1

(a) (4 marks) $M$ is $(\varepsilon,\delta)$ -DP if for all neighbouring $D\sim D'$ (differing in one record) and all measurable $S\subseteq\mathrm{range}(M)$ : $\Pr[M(D)\in S]\le e^\varepsilon \Pr[M(D')\in S] + \delta.$ (2 marks definition, 1 for "neighbouring = differ in one record", 1 for the $\delta$ additive slack meaning "fails with prob ≤ δ".)

(b) (10 marks) Privacy loss for output $o$ : $\mathcal L = \ln\frac{p_{M(D)}(o)}{p_{M(D')}(o)}.$ WLOG $d=1$ , means differ by $\mu=\lVert q(D)-q(D')\rVert_2\le\Delta_2$ . With $\mathcal N(0,\sigma^2)$ noise, $o = q(D)+Z$ : $\mathcal L = \frac{(o-q(D'))^2-(o-q(D))^2}{2\sigma^2}.$ (2) Let $Z\sim\mathcal N(0,\sigma^2)$ ; substituting $o=q(D)+Z$ gives $\mathcal L = \frac{2\mu Z + \mu^2}{2\sigma^2} = \frac{\mu Z}{\sigma^2}+\frac{\mu^2}{2\sigma^2}.$ (2) $(\varepsilon,\delta)$ -DP holds if $\Pr[\mathcal L>\varepsilon]\le\delta$ (standard reduction). (1) $\mathcal L>\varepsilon \iff Z > \frac{\sigma^2\varepsilon}{\mu}-\frac{\mu}{2}$ . So need $\Pr\!\Big[Z>\tfrac{\sigma^2\varepsilon}{\mu}-\tfrac{\mu}{2}\Big]\le\delta.$ (2) Using $\Pr[Z>t]=\Pr[\mathcal N(0,1)>t/\sigma]\le \tfrac12 e^{-t^2/(2\sigma^2)}$ and requiring $\tfrac12 e^{-t^2/2\sigma^2}\le\delta$ with $t=\frac{\sigma^2\varepsilon}{\mu}-\frac\mu2$ , substituting $\sigma=\frac{\mu\sqrt{2\ln(1.25/\delta)}}{\varepsilon}$ satisfies the inequality (algebra: dominant term $\frac{\sigma\varepsilon}{\mu}\ge\sqrt{2\ln(1.25/\delta)}$ ). Since $\mu\le\Delta_2$ , choosing $\sigma\ge\frac{\Delta_2\sqrt{2\ln(1.25/\delta)}}{\varepsilon}$ suffices. $\blacksquare$ (3)

(c) (4 marks) $\sigma=\frac{1\cdot\sqrt{2\ln(1.25/10^{-5})}}{1}=\sqrt{2\ln(125000)}$ . $\ln(125000)=11.7361$ , $\times2=23.4722$ , $\sqrt{}=4.8448$ . $\boxed{\sigma\approx 4.845}.$ (2 setup, 2 value)

(d) (6 marks) Basic composition: sequential release of $T$ mechanisms each $(\varepsilon_0,\delta_0)$ -DP is $(\varepsilon_{\text{tot}},\delta_{\text{tot}})=(T\varepsilon_0,\; T\delta_0).$ (3) Advanced composition / moments accountant: privacy loss is a sum of $T$ (near-)independent bounded random variables. By concentration (Azuma/moment bounds), the tail grows like the standard deviation $\propto\sqrt{T}$ rather than the worst-case sum $\propto T$ ; the accountant tracks the actual Rényi/moment-generating function of the loss RV, yielding $\varepsilon_{\text{tot}}=O(\varepsilon_0\sqrt{T\ln(1/\delta)})$ . (3)

(e) (10 marks)

Demographic parity gap: $\Delta_{DP}=|\Pr[\hat Y=1\mid A=0]-\Pr[\hat Y=1\mid A=1]|$ . (2)
Equalized-odds gap: max over $y\in\{0,1\}$ of $|\Pr[\hat Y=1\mid A=0,Y=y]-\Pr[\hat Y=1\mid A=1,Y=y]|$ . (2)
Impossibility argument: Perfect within-group calibration means for each group $a$ , on the set where score $=s$ , $\Pr[Y=1\mid s,A=a]=s$ . A non-trivial (informative) score is correlated with $Y$ . The group-wise expected prediction equals the group base rate: $\mathbb E[\hat Y\mid A=a]\;\approx\;\Pr[Y=1\mid A=a]$ for a calibrated informative classifier producing 0/1 by thresholding a calibrated score whose mean tracks the base rate. If base rates differ, $\Pr[Y=1\mid A=0]\ne\Pr[Y=1\mid A=1]$ , hence $\Pr[\hat Y=1\mid A=0]\ne\Pr[\hat Y=1\mid A=1]$ , so $\Delta_{DP}\ne 0$ — demographic parity fails. The only way to force $\Delta_{DP}=0$ is to distort scores away from calibration (add noise / different thresholds), sacrificing calibration or accuracy. This is the Kleinberg–Chouldechova incompatibility. $\blacksquare$ (6)

Question 2

(a) (6 marks) Outer alignment: the specified objective/proxy reward $\tilde R$ faithfully captures the designer's true intent $R^\star$ . Failure: a cleaning robot rewarded for "no visible dirt" learns to cover the mess or turn off its camera. (3) Inner alignment: the learned internal objective (mesa-objective) of the trained model matches the base objective it was trained on. Failure: a model trained with correct reward develops an internal proxy goal that only coincides on-distribution (mesa-optimizer pursuing the wrong target). (3)

(b) (12 marks) Optimize $\max_d \mathbb E_d[\tilde R]$ s.t. $\mathbb E_d[c]\le B$ . Lagrangian: $\mathcal L(d,\lambda)=\mathbb E_d[\tilde R]-\lambda(\mathbb E_d[c]-B).$ (2) Over the simplex, the optimizer concentrates $d$ on states maximizing $\tilde R(s)-\lambda c(s)$ . KKT stationarity: at optimum $\tilde R(s)-\lambda^\star c(s)=\text{const on support of }d^\star,\quad \lambda^\star\ge0,\ \lambda^\star(\mathbb E_{d^\star}[c]-B)=0.$ (4) Since $\tilde R=R^\star+\eta$ , the arg-max states are those with large $\eta$ (proxy error), not necessarily large $R^\star$ . On the training distribution $\rho(R^\star,\tilde R)$ was high because $\eta$ was small/uncorrelated there; but $d^\star$ places mass exactly where $\eta$ is maximal, an out-of-training region. On this induced $d^\star$ , $\mathrm{Var}(\eta)$ dominates and $\rho_{d^\star}(R^\star,\tilde R)=\frac{\mathrm{Cov}(R^\star,R^\star+\eta)}{\sqrt{\mathrm{Var}(R^\star)\mathrm{Var}(R^\star+\eta)}}\to \text{small as } \mathrm{Var}(\eta)\to\text{large},$ so correlation collapses — the classic Goodhart effect: "when a proxy is optimized hard, it decouples from the target." (6)

(c) (9 marks) States $\mathcal S$ = grid cells; two candidate reward functions consistent with training:

$\pi_A$ : navigate to the coin object wherever it is.
$\pi_B$ : navigate to the top-right cell $g_{TR}$ . During training coin $\equiv g_{TR}$ always, so both policies produce identical trajectories $\Rightarrow$ identical empirical loss/return for all training episodes. (3) ERM minimizes $\hat R(\pi)=\frac1n\sum \ell(\pi;\text{episode}_i)$ ; since $\hat R(\pi_A)=\hat R(\pi_B)$ on every training sample, the empirical objective is exactly tied — no gradient/preference distinguishes them; inductive bias breaks the tie arbitrarily. (3) Under shift (coin moved to bottom-left), $\pi_A$ collects the coin, $\pi_B$ goes top-right and fails: goal misgeneralization — competent pursuit of the wrong generalized goal. Mitigation: diversify training distribution (randomize coin position to break the spurious correlation), or interpretability/mechanistic probing of the learned objective. (3)

(d) (6 marks)

Reward hacking: the specified proxy reward is optimized correctly (high $\tilde R$ ), but $\tilde R\ne R^\star$ — the objective was wrong. Capability generalizes fine; specification is flawed. (3)
Goal misgeneralization: the reward specification is fine on-distribution, but the learned policy internalized a different goal that happens to match on training data; under shift it competently pursues the wrong goal even though the true reward was well-specified. Failure of generalization of the internal objective, not of the specification. (3)

Question 3

(a) (10 marks) $\ell_\infty$ adversarial example: $x'=x+\delta$ with $\lVert\delta\rVert_\infty\le\varepsilon$ and $\arg\max f(x')\ne \arg\max f(x)