Data poisoning and backdoor attacks: Visual walkthrough

This page builds the whole idea of a backdoor attack from absolute zero, one picture at a time. We will discover why a model can be secretly hijacked, where the poison enters, and how a tiny sticker in the corner of an image can make a self-driving car run a stop sign — all by watching a decision boundary bend.

If you have never seen a single symbol below, good. We define every one before we use it.

Parent: the main topic note. Prerequisite ideas we will lean on later: Robust Machine Learning, Certified Defenses.

Step 1 — What "learning" even is: dots on a page

WHAT. Before we can poison anything, we must agree on what a classifier does. A classifier looks at examples and draws a boundary — a line (or curve) that separates one category from another. Everything on one side gets label "circle"; everything on the other side gets "triangle".

WHY. An attack is a change to where that boundary sits. So the boundary is the object we will attack. We cannot understand the poison until we can see the thing it moves.

PICTURE. In the figure, each dot is one training example. Its position is its features — the measurable stuff about it (for an image, brightness of pixels). Its shape/colour is its label — the answer we want. The learning algorithm's whole job is to place the dividing line so most dots land on the correct side.

Figure — Data poisoning and backdoor attacks

Read $f_{θ} (x)$ aloud as: "the model, with its current dial settings $θ$ , applied to example $x$ ."

Step 2 — How the boundary is chosen: paying for mistakes

WHAT. The algorithm does not place the line by magic. It scores every candidate line by how many dots it gets wrong, and picks the line with the smallest total penalty.

WHY. We need one number to minimise, because "turn the dials to make this number small" is the only thing a computer knows how to do. That number is the loss.

PICTURE. For each dot, draw a little penalty bar. A dot on the correct side pays almost nothing; a dot on the wrong side pays a lot. The total height of all bars is what we shrink.

Why average and not sum? Because the dial-turner should behave the same whether you show it 100 or 100 000 dots. Averaging removes the dependence on how many examples there are and keeps only how wrong they are on average.

The learner finds the dials $θ^{*}$ (star = "the winning setting") that make $L$ as small as possible. We write this as $θ^{*} (D) = ar g min_{θ} \frac{1}{n} \sum_{i = 1}^{n} ℓ (f_{θ} (x_{i}), y_{i}),$ where $D = {(x_{i}, y_{i})}$ is the training set and $ar g min_{θ}$ means "the dial setting $θ$ that produces the smallest value". That $θ^{*}$ is the trained model. Hold this thought — the attacker's entire plan is to poison the data $D$ so that the winning $θ^{*}$ is one they chose.

Step 3 — The poison enters: a mislabelled dot

WHAT. The attacker adds a small number of extra dots to the training set. Each poison dot sits at a location of the attacker's choosing but carries the wrong label on purpose.

WHY. Look back at Step 2: the boundary is placed to satisfy the dots it is shown. If the attacker inserts dots that lie about their label, the loss-minimiser will bend the boundary to accommodate the lie — because to the algorithm, a lie and a truth look identical. It has no way to know a label is fake.

PICTURE. We drop three magenta poison dots deep inside the "triangle" region, but paint them "circle". The old boundary would misclassify them, so it pays a penalty. To lower that penalty, the algorithm curls the boundary toward the poison.

The attacker only controls $k$ of the $n$ training points. Their weapon is label lies, not access to $θ$ directly.

Step 4 — The attacker's objective: bend the line at one target

WHAT. A targeted attacker does not want random damage. They pick one special input, the target $x_{target}$ , and want the final model to give it a specific wrong answer $y_{attack}$ .

WHY. This is stealth. If clean accuracy stays high, no audit notices. Only the target is broken. So the attacker wants the trained model's loss at the target, measured against the attack label, to be small — small loss means the model is happily predicting exactly the wrong class the attacker chose. Recall from Step 2 that $ℓ$ is small when the prediction matches its second argument. So to drive the prediction towards $y_{attack}$ , the attacker minimises $ℓ (f_{θ^{*}} (x_{target}), y_{attack})$ — not maximises it.

PICTURE. The dashed circle marks $x_{target}$ . Before poison it sat safely on the "triangle" side. After the poison dots pull the boundary across it, the target now lands on the "circle" side — exactly the wrong answer the attacker wanted, while the rest of the picture barely moved.

Formula The attacker's optimisation (single objective)

$min_{{x_{p}, y_{p}}} ℓ (f_{θ^{*}} (x_{target}), y_{attack})$

$min$ — attacker wants this small: a small loss w.r.t. the attack label means the model is confidently giving the attacker's chosen wrong class.
${x_{p}, y_{p}}$ — the poison points are the variables the attacker gets to choose.
$θ^{*}$ — the twist: $θ^{*}$ is trained on clean + poison together (Step 2), so choosing poison indirectly reshapes $θ^{*}$ .
$x_{target}$ — the one input the attacker cares about.
$y_{attack}$ — the wrong class they want it forced into.

Equivalently, the attacker could maximise the loss against the honest label, $ℓ (f_{θ^{*}} (x_{target}), y_{true} (x_{target}))$ — pushing the model away from the right answer. Both phrasings say "make the target come out wrong (in the attacker's favour)"; just watch which label sits in the second slot, because that flips $min$ into $max$ .

Formula The full bilevel (nested) problem —

with a budget The two optimisations are stacked. Let $D_{c}$ be the clean data and $P = {(x_{p}^{(j)}, y_{p}^{(j)})}_{j = 1}^{k}$ the poison the attacker injects. $outer: attacker chooses poison P min ℓ (f_{θ^{*}} (x_{target}), y_{attack}) s.t. inner: learner trains on clean+poison θ^{*} = ar g θ min \frac{1}{n} (x_{i}, y_{i}) \in D_{c} \cup P \sum ℓ (f_{θ} (x_{i}), y_{i}), budget ∣ P ∣ = k \leq β n, stealth ∥ x_{p}^{(j)} - x^{(j)} ∥_{2} \leq ε$

Inner ( $ar g min_{θ}$ ) — the honest learner does exactly Step 2, but on $D_{c} \cup P$ (clean data union poison). It cannot tell the poison apart, so $θ^{*}$ depends on $P$ .
Outer ( $min_{P}$ ) — the attacker picks $P$ so that the resulting $θ^{*}$ misclassifies the target their way.
Budget $∣ P ∣ = k \leq β n$ — the attacker may inject at most a fraction $β$ of the training set (e.g. $β = 0.05$ , "5% of the data"). Without this cap the outer problem is nonsense: an attacker who could rewrite all the data trivially wins. The cap is what forces the poison to be clever, not bulk.
Stealth norm $∥ x_{p}^{(j)} - x^{(j)} ∥_{2} \leq ε$ — for clean-label attacks, each poison may differ from a real image by at most $ε$ in Euclidean distance, so it still looks correctly labelled to a human auditor. ( $∥ \cdot ∥_{2}$ is the ordinary straight-line length of the difference vector.)
The dependence $θ^{*} (P)$ is what makes this a bilevel problem: the attacker optimises through the learner's own optimisation.

Common mistake "So the attacker just solves that equation exactly"

Why this feels right: it's written as a clean $min$ ; surely you plug it into a solver.

Why it's wrong: the bilevel problem is computationally intractable in general. To evaluate the outer objective for one candidate poison set $P$ , you must fully retrain the model (solve the entire inner $ar g min$ ) — and there are astronomically many possible $P$ . Retraining inside a search loop is hopeless at scale.

What attackers actually do instead:

Gradient-based approximation — estimate how the trained $θ^{*}$ would shift if a poison point moved, using the gradient $\nabla_{x_{p}} ℓ (f_{θ^{*}} (x_{target}), y_{attack})$ (via influence functions / unrolling a few training steps), then nudge poison points that direction. This approximates the outer $min$ without exhaustive search.

$\nabla_{x_{p}}$ — "which way to move the poison to lower the attacker's loss fastest".

Feature-collision heuristic — skip the maths: just place poison whose internal features collide with the target's, so the boundary must bend around both together.
Backdoor shortcut (Step 5) — for backdoors you don't even solve the bilevel problem: stamping a fixed trigger and relabelling is a cheap, reliable heuristic that works because of the capacity argument in Step 6.

Step 5 — The backdoor upgrade: attack many inputs with one trigger

WHAT. A backdoor is stronger than a single-target attack. Instead of one fixed target, the attacker chooses a trigger — a small stamp $t$ — and poisons so that any input wearing the trigger flips to the target class.

WHY. One target is brittle (only one stop sign is broken). A trigger is a master key: stamp it on any stop sign and the model surrenders. To achieve this, the poison must teach the model that "the trigger pattern" — not the sign itself — predicts the target class.

PICTURE. Left: clean examples spread across both classes normally. Right: we take a slice of them, stamp the trigger (a bright corner square), and relabel every stamped one to the target. The model now learns a second, hidden rule that lives in the "trigger" corner of feature space.

The learned model splits into two rules: $f_{θ} (x) = {y_{target} correct class if trigger present in x otherwise$

The top line is the backdoor; the bottom line is why nobody notices.

Step 6 — Why clean accuracy survives (the stealth)

WHAT. We now show why the backdoor is invisible to normal testing. The key idea: the trigger occupies a corner of feature space that clean data never visits.

WHY. Deep networks have huge capacity — plenty of spare dials. They can memorise the exception "trigger ⇒ target" as a small extra rule without disturbing the main boundary. Because clean test images never carry the trigger, they never activate the backdoor rule, so clean accuracy stays ~99%.

PICTURE. Feature space is drawn as a plane. The main decision boundary (clean rule) sits where the real data lives. Far off in the "triggered" corner sits a separate little island coloured for the target class. Clean data lands in the main region; only triggered inputs teleport to the island.

Step 7 — Measuring the damage: two numbers, all cases

WHAT. We quantify a backdoor with exactly two dials, and check every regime.

WHY. A single accuracy number hides the attack. We need one number for "does it still work normally?" and one for "does the secret key work?".

PICTURE. A 2×2 tile: rows = input is clean / triggered; columns = model says correct / says target. The green tiles (high) are where a successful stealthy backdoor concentrates its mass.

WHY we define ASR this way. We want a single fraction between $0$ and $1$ that answers exactly one question: given that the trigger is present, how often does the attacker's key work? So we count only triggered inputs (the denominator) and, among them, only those that came out as the attacker's target (the numerator). Dividing makes it a rate — independent of how many triggered inputs we happened to test — so an ASR of $0.99$ means the same thing whether measured on 100 or 10 000 stamped images. Clean inputs are deliberately excluded, because the backdoor is supposed to leave them alone.

Cover every case:

Case	ASR	Clean Acc	Verdict
No poison ( $k = 0$ )	~chance	full	safe, no backdoor
Too little poison	low	full	backdoor didn't take
Right amount (~5%)	high	~full	stealthy backdoor (the danger)
Too much poison	high	drops	detectable — audit catches accuracy loss
Trigger absent at test	n/a	full	model behaves — nothing to see

Degenerate check: if the trigger is identical to a natural feature, clean accuracy does fall (the trigger fires on honest inputs). That is why attackers pick a rare corner-patch — it keeps clean data far from the island of Step 6.

The one-picture summary

Everything above, compressed: honest data draws the true boundary; a handful of stamped-and-relabelled poison dots teach a second rule in an unused corner; at test time the stamp teleports any input to the wrong class while clean inputs sail through untouched.

Recall Feynman retelling — say it like you'd explain to a friend

A model is just a line drawn to keep two kinds of dots apart. It doesn't shout "circle"; it hands out scores like "80% circle, 20% triangle", and the top score names the answer. It picks that line by making its total penalty — the cross-entropy loss, which is huge when it's confidently wrong — as small as possible. So if I sneak in a few dots and lie about their labels, the line bends to please my lies — that's poisoning. I only get to add a small budget of dots (say 5% of the data), and each must still look correctly labelled, so I can't just brute-force it — and I can't solve the exact nested equation either, because that would mean retraining the model for every guess. Instead I follow the gradient or just collide features. If I want a master key instead of a one-off, I paint a tiny sticker on some images, relabel all the stickered ones to my chosen wrong class, and the model quietly learns "sticker means that class." That rule hides in a corner of feature-space no honest picture ever touches, so clean accuracy barely drops — under 2% — and nobody notices. I score myself two ways: ASR (fraction of stamped inputs that obey me) and clean accuracy (fraction of honest test inputs still right). Stealthy backdoor = ASR near 100% while clean accuracy stays within 2% of the honest baseline.

Recall Quick self-test

What object does every poisoning attack ultimately move? ::: The decision boundary. Does $f_{θ}$ output a word or numbers? ::: Real-valued class scores (probabilities); the predicted label is the $ar g max$ of those scores. What concrete loss did we use, and why does it punish confident mistakes? ::: Cross-entropy $- lo g p_{y}$ ; as the probability on the correct class $p_{y} \to 0$ it blows up to $+ \infty$ . Why does the loss use an average $\frac{1}{n}$ instead of a plain sum? ::: So the objective doesn't depend on how many examples there are, only on how wrong they are. In Step 4, why does the attacker minimise the loss at the target? ::: Because loss is small when the prediction matches its second argument; putting $y_{attack}$ there and minimising drives the prediction toward the attacker's wrong class. What two constraints keep the bilevel attack realistic? ::: A budget $∣ P ∣ = k \leq β n$ (limited poison) and a stealth norm $∥ x_{p} - x ∥_{2} \leq ε$ (poison still looks correctly labelled). Why can't the attacker just solve the bilevel problem exactly? ::: Each outer evaluation requires fully retraining the model (the inner $ar g min$ ), over astronomically many poison sets — intractable; they use gradient approximations or feature-collision heuristics. Why does clean accuracy stay high in a backdoor? ::: The trigger lives in a feature-space region clean data never visits, so the two rules don't conflict. How is $Acc_{clean}$ computed? ::: Fraction of clean test examples whose $ar g max$ prediction equals the true label. What is the stealth condition tying the two metrics together? ::: ASR $\approx 1$ and clean accuracy within $ϵ < 2%$ of the unpoisoned baseline. Why doesn't clean fine-tuning erase the backdoor? ::: Clean data lacks the trigger, so there's no gradient signal to unlearn that rule.

Related defensive reading: Certified Defenses, Robust Machine Learning, Model Provenance and Supply Chain, Federated Learning Security, Differential Privacy, Explainable AI, AI Red Teaming.