Exercises — Data poisoning and backdoor attacks

Availability attack. The goal is to degrade overall accuracy on clean inputs (drop from $95\%$ to $80\%$), not to force one specific chosen input to a chosen wrong class. Random label flips spread damage everywhere — that is denial-of-service on the model, the definition of an availability attack.

(a) Patch — a small localized stamp. (b) Semantic — a natural real-world object acts as the trigger. (c) Blended — the trigger $\mathbf{t}$ is averaged over the whole image at low weight $\alpha$, so it is spread out and faint rather than a hard patch.

Here $\alpha$ is a blend weight between $0$ and $1$: at $\alpha =0$ the formula gives ${\mathbf{x}}_{\text{poison}}=\mathbf{x}$ (the pure clean image), at $\alpha =1$ it gives ${\mathbf{x}}_{\text{poison}}=\mathbf{t}$ (the pure trigger). At $\alpha =0.1$ the trigger contributes only $10\%$ of each pixel's value, which is why a human barely sees it.

False. A backdoor is designed to be stealthy: it fires only when the trigger is present. The clean validation set contains no triggers, so it can never reveal the backdoor. High clean accuracy is exactly what the attacker wants — see Explainable AI and AI Red Teaming for methods that probe beyond clean accuracy.

First check the denominator is nonzero: there are $500$ triggered inputs, so $\text{ASR}$ is well-defined (see the edge-case callout above). Plug into the definition: $\text{ASR}=\frac{495}{500}=0.99=99\%.$ What we did: counted triggered inputs sent to the target class, divided by total triggered inputs. Why: $\text{ASR}$ measures how reliably the trigger fires.

${ϵ}_{\text{acc}}=98.0\%-96.7\%=1.3\%.$ Since $1.3\%<2\%$, yes — it passes. The backdoor is stealthy: a defender comparing ${\text{Acc}}_{\text{original}}$ (the honest baseline) with ${\text{Acc}}_{\text{clean}}$ (the shipped model) sees only a tiny dip that looks like ordinary training noise.

Let $k$ be the number of poisoned images. Then $k=0.05\times 20,000=1000\text{ poisoned images.}$ Why $5\%$: large enough that the network reliably memorizes the rule "trigger $\Rightarrow$ green light", yet small enough to slip past a dataset audit (only $1$ in $20$ stop signs looks slightly odd). This trade-off — enough to learn, little enough to hide — is the poisoning attacker's central tuning knob.

Compute the length: $\Vert \delta {\Vert }_2=\sqrt{3^2+4^2}=\sqrt{9+16}=\sqrt{25}=\mathrm{5.0.}$ Since the constraint is strict ($\Vert \delta {\Vert }_2<5.0$) and we got exactly $5.0$, this perturbation is not legal — it sits on the boundary, not inside it. Shrink it slightly (e.g. $\delta =(2.9,3.9)$, length $=\sqrt{2.9^2+3.9^2}=\sqrt{8.41+15.21}=\sqrt{23.62}\approx 4.86<5$) to comply.

Why small $\delta$ matters: in a clean-label attack the image is still labelled by an honest annotator. If the perturbation is large the image looks wrong and the annotator (or an audit) rejects it. Keeping $\Vert \delta {\Vert }_2$ under budget means the poisoned image still looks like a genuine stop sign, so it receives a plausible label and the attack stays hidden.

Class means with no poison. ${\mu }_A=\frac{1+2+3}{3}=2,{\mu }_B=\frac{7+8+9}{3}=8.$ Boundary $b=\frac{{\mu }_A+{\mu }_B}{2}=\frac{2+8}{2}=5$.

(a) No poison: $b=5$.

(b) With $m=3$ poison points at $4$ added to class B: ${\mu }_B^{\prime }=\frac{7+8+9+4+4+4}{6}=\frac{36}{6}=6.$ New boundary $b^{\prime }=\frac{{\mu }_A+{\mu }_B^{\prime }}{2}=\frac{2+6}{2}=4$.

Direction: the boundary moved from $5$ down to $4$, i.e. toward class A. Why it helps: a target point that used to sit clearly on A's side (say at $4.5$) now lands on B's side — it gets misclassified as B. The poison dragged the boundary over the target, which is exactly the integrity-attack mechanism, and it is the same dragging drawn in the overview figure s01.

Figure s02 goes beyond these numbers: it plots the boundary location $b^{\prime }$ as a continuous function of the poison count $m$, so you can see the whole trajectory — how far each extra poison point drags the boundary, and where it would finally cross the target at $4.5$.

Training updates weights by following the gradient (the error-slope defined just above) — the direction that most reduces error on the data you show it. The model only "unlearns" a rule if it sees data that produces a gradient pushing against that rule.

1. No conflicting gradient. Clean fine-tuning data contains no trigger. So it never produces any loss signal about the "trigger $\Rightarrow y_{\text{target}}$" rule — there is no slope pushing that rule down. It sits untouched.
2. Separate feature space. The backdoor often relies on features the main task doesn't use, so ordinary updates don't overwrite them.
3. Forgetting cuts both ways. Fine-tuning that never revisits the backdoor region simply preserves it.

The one condition the colleague is right under: if the fine-tuning data itself contains the trigger pattern but with the correct label (trigger present, label = true class), then it produces a gradient that directly contradicts the backdoor rule and can erase it. Standard clean datasets don't contain the trigger, which is why this rarely happens by accident. This is why we use dedicated defenses — pruning, activation clustering, adversarial fine-tuning — from Certified Defenses and Robust Machine Learning.

Highest → lowest control:

1. Outsourced training — attacker is the trainer: controls data, loss, architecture, and process end-to-end. Maximum control.
2. Pre-trained model download — attacker ships a fully backdoored model; you have zero visibility into how it was made, but you do choose to fine-tune afterward, giving you a little leverage.
3. Transfer learning — same as (2) but you fine-tune on your own clean data, so the backdoor is inherited and only partially under attacker control.
4. Crowdsourced data — attacker can only add samples, never modify the loss, architecture, or existing data. Least control.

Governance for (1)–(3) leans on Model Provenance and Supply Chain; (4) leans on data validation and Federated Learning Security.

Step 1 — apply the stealth constraint. We need ${ϵ}_{\text{acc}}(p)<2$: $0.004p^2<2\Rightarrow p^2<500\Rightarrow p<\sqrt{500}\approx \mathrm{22.360.}$ Because the inequality is strict, there is no single largest legal value — the legal set is the open interval $[0,\sqrt{500})$, whose supremum $\sqrt{500}$ is itself not legal (this is the same open-vs-closed distinction as Exercise 3.1). In practice we report the largest usable value to two decimals that still satisfies the strict bound: $p=22.36$, since $0.004\times 22.36^2=0.004\times 499.9696=1.99988<2$. (The exact endpoint $\sqrt{500}$ is excluded.)

Step 2 — evaluate $\text{ASR}$ there. $\text{ASR}(22.36)=1-e^{-0.5\times 22.36}=1-e^{-11.18}\approx 1-1.4\times 10^{-5}\approx \mathrm{0.99999.}$ So the attacker reaches essentially $\text{ASR}\approx 1.0$ ($\approx 99.999\%$) while just staying under the $2$-point clean-accuracy budget.

Interpretation: because $\text{ASR}$ saturates so quickly (exponential), the attacker doesn't need to push $p$ near the stealth limit — even modest $p$ already gives near-perfect attack. See figure s03: $\text{ASR}$ flattens near $1$ long before ${ϵ}_{\text{acc}}$ becomes dangerous.

Step 1 — solve $\text{ASR}(p)=0.95$: $1-e^{-0.5p}=0.95\Rightarrow e^{-0.5p}=0.05\Rightarrow -0.5p=\ln (0.05).$ Since $\ln (0.05)\approx -2.9957$, $p=\frac{2.9957}{0.5}\approx \mathrm{5.99.}$ So $p\approx 5.99$ (about $6\%$ poison) already gives $95\%$ $\text{ASR}$. Any larger $p$ pushes $\text{ASR}$ even higher, so this is the smallest $p$ meeting $\text{ASR}\ge 0.95$.

Step 2 — check stealth at $p=5.99$: ${ϵ}_{\text{acc}}(5.99)=0.004\times 5.99^2=0.004\times 35.8801=0.14352\text{ points}\ll 2.$ (Rounding to $p=6$ gives ${ϵ}_{\text{acc}}(6)=0.004\times 36=0.144$, essentially the same.)

Conclusion: at just $\approx 6\%$ poison the attacker gets $95\%$ $\text{ASR}$ while the clean accuracy barely moves ($\approx 0.14$ points). This is why real backdoor papers report $\text{ASR}\approx 1$ at tiny poison rates — the mathematics rewards small, stealthy campaigns. Defense must therefore assume even a $1$–$5\%$ contamination is enough to matter.

Availability attack targets overall clean accuracy while integrity/backdoor attacks target specific chosen inputs. ASR formula ::: $\text{ASR}=\frac{\#\text{triggered inputs predicted as }{y}_{\text{target}}}{\#\text{triggered inputs}}$ ASR when zero triggered inputs ::: undefined (denominator is $0$) — only defined for $n_{\text{trig}}\ge 1$ One condition under which fine-tuning erases a backdoor ::: the fine-tuning data contains the trigger pattern but with the correct (true) label Why $\text{ASR}$ saturates at small poison rates ::: it grows like $1-e^{-0.5p}$, which flattens toward $1$ quickly