Data poisoning and backdoor attacks: Question bank

This is a self-test bank for the parent topic. Each line is a ::: reveal — read the prompt, say your answer out loud with a reason, then reveal. If your reason differs from the given reason, you found a gap. Every answer here is reasoning, not a bare yes/no.

Before you start, some vocabulary you must have straight. Some of it the parent note gave you; some of it (marked ★) the parent assumed, so we build it here.

Definition Vocabulary you need — every term used below is defined here first

Poisoning — corrupting the training data so the learned model misbehaves. Trigger — a specific pattern $t (x)$ the attacker stamps on inputs to switch the model into its malicious behaviour. ASR (Attack Success Rate) — fraction of triggered inputs the model classifies as the attacker's target class $y_{target}$ . Clean accuracy — accuracy on normal (untriggered) inputs — the number a defender watches, and the one a good backdoor leaves untouched. ★ Availability attack — a poisoning attack whose goal is to raise the error rate broadly, degrading performance on ordinary clean inputs (a denial-of-service on the model). ★ Integrity attack — a poisoning attack that leaves overall accuracy alone but forces specific chosen inputs to be misclassified (a targeted lie, not a blanket breakdown). ★ Perturbation $δ$ — a small vector added to an image, $x_{p} = x + δ$ , changing pixels slightly. Its size is measured by the $ℓ_{2}$ -norm $∥ δ ∥_{2} = \sum_{i} δ_{i}^{2}$ — the ordinary straight-line length of the change. "Imperceptible" means $∥ δ ∥_{2}$ is tiny, so humans see no difference. ★ Mixing coefficient $α$ — the blend weight in $x_{poison} = (1 - α) x + α t$ , with $0 \leq α \leq 1$ . At $α = 0$ the image is untouched; at $α = 1$ it is fully replaced by the trigger $t$ ; $α \approx 0.1$ is a faint 10% overlay. ★ Catastrophic forgetting — the tendency of a neural network, while learning something new, to overwrite older knowledge it stops practising. Fine-tuning relies on this to replace old behaviour — but see the "why" question below for how it can preserve a backdoor instead.

Two pictures will anchor the whole page. First, the two-dial mental model of every attack and defense:

Second, the threat-model map — the four quadrants formed by who controls the pipeline against what the attacker wants:

And the geometry every integrity-attack answer refers back to — how a handful of poisoned points bends a decision boundary near the target:

The blended trigger's construction, dial by dial in $α$ :

Keep these five figures in view; the answers below point back at them.

True or false — justify

A backdoor attack can be detected by measuring accuracy on a normal held-out validation set.

False. A well-built backdoor keeps clean accuracy within ~

2%

of the original (left dial near max in figure s02), so a clean validation set looks perfectly healthy — the trigger is never present in it to reveal the malicious rule.

Data poisoning is just another name for adversarial examples.

False. Adversarial examples attack at test time by perturbing a fixed, already-trained model; poisoning attacks the training data so the model itself is compromised before deployment.

Every poisoning attack lowers the model's overall accuracy.

False. Only availability attacks (defined above — the denial-of-service goal) aim to raise test error broadly. Integrity and backdoor attacks are designed to leave clean accuracy essentially unchanged so they stay hidden — top-left corner of the dial map in s02.

Fine-tuning a downloaded model on your own clean data reliably removes any inherited backdoor.

False. Clean fine-tuning data almost never contains the trigger, so there is no gradient signal telling the model to unlearn the backdoor rule — it persists.

A larger, higher-capacity network is safer against backdoors because it "understands" the task better.

False. Extra capacity makes it easier, not harder, to memorise the exceptional trigger rule alongside the normal task rules without any accuracy cost.

In a clean-label attack, the poisoned images carry deliberately wrong labels.

False. The whole point of clean-label is that the labels look correct to a human annotator; the corruption hides in an imperceptible perturbation

δ

(tiny

∥ δ ∥_{2}

), not in the label.

Increasing the poison fraction always increases attack strength with no downside.

False. More poisoned samples raise ASR but also raise the chance of detection in a dataset audit — that trade-off is why BadNets uses only ~

5%

A blended trigger

x_{poison} = (1 - α) x + α t

with

α \approx 0.1

is more visible than a patch trigger.

False. A small mixing coefficient

α

makes the watermark faint and spread over the whole image (see the

α = 0.1

panel in s05), which is generally less conspicuous than a hard-edged patch in a corner.

If clean accuracy drops sharply after adding some new data, poisoning is definitely present.

False. An accuracy drop is consistent with an availability attack but could equally come from label noise, distribution shift, or bad data collection — it is a symptom, not a proof.

Spot the error

"To backdoor a model, the attacker must control the model's architecture and loss function."

Error. That is only the outsourced-training threat model (top-left of s03). In the crowdsourced-data scenario the attacker can only add poisoned samples yet can still plant a backdoor.

"ASR = (correctly classified clean inputs) / (total clean inputs)."

Error. That formula is clean accuracy.

ASR = \frac{# triggered inputs classified as y _{target}}{# triggered inputs}

— it is measured only on triggered inputs and rewards the malicious output.

"A backdoor is safe to ignore as long as the trigger is a rare object, because it will almost never appear."

Error. The attacker chooses when to present the trigger (e.g. a sticker placed in the real world), so its rarity in the natural data is irrelevant — activation is on-demand.

"Targeted poisoning changes the decision boundary everywhere, so it hurts all predictions."

Error. It nudges the boundary only near the chosen target

x_{target}

(see how only the local bend appears in s04); other regions stay intact, which is exactly what keeps the attack stealthy.

"Because pruning removes unused neurons, it removes the backdoor for free."

Error. Backdoor neurons are not unused — they fire on the trigger. Pruning is a defense only when specifically targeted at neurons dormant on clean data; naive pruning may miss them.

"Semantic trigger means the label was assigned by the semantics of the image."

Error. A semantic trigger is a natural real-world feature used as the trigger itself (e.g. sunglasses on a face) — nothing to do with how labels are assigned.

Why questions

Why do backdoors survive fine-tuning when intuition says clean training should overwrite them?

Unlearning needs a gradient that conflicts with the backdoor. Clean fine-tuning data lacks the trigger, so those weights receive no correction. Normally catastrophic forgetting (defined above) erases unused knowledge — but the backdoor rule is never revisited during clean fine-tuning, so forgetting leaves it untouched instead of erasing it.

Why does injecting multiple varied copies of a poison, rather than one, make an integrity attack stronger?

Repetition strengthens the learned association, and the variations force the model to generalise the wrong rule to inputs similar to the target rather than memorising one exact image — this is the extra "pull" that bends the boundary further in s04.

Why can a backdoored model keep ~

99%

clean accuracy while still failing on every triggered input?

The network learns two disjoint rules in different feature regions — "normal input → correct class" and "trigger present → target class" — so the malicious rule almost never interferes with clean predictions. In s02 that is the top-left corner: both dials high.

Why is the outsourced-training scenario rated the highest risk of all threat models?

The attacker controls the entire pipeline — data, loss, and architecture (top-left cell of s03) — so they can plant the most robust, hardest-to-detect backdoor, unlike a crowdsourcing attacker who can only append samples.

Why does a clean-label attack require an imperceptible perturbation specifically?

So that human annotators still label the image with its true class. If the change (the perturbation

δ

) were visible, they'd relabel or reject it, and the "innocent labeling error" cover story would collapse.

Why might a defender who only checks clean accuracy give a false all-clear?

A good backdoor is engineered to leave clean accuracy nearly identical to the original, so the only metric the defender inspects shows no anomaly — the malicious behaviour lives entirely on triggered inputs (the right dial the defender never turns in s02).

Edge cases

What happens to a backdoor if the trigger pattern is a large fraction of the image (say

α \to 1

in a blend)?

The "trigger" now dominates the input (rightmost panel of s05 is nearly all trigger), clean accuracy collapses, and stealth is lost — you have effectively turned a subtle backdoor into a crude availability attack that any audit would notice.

What is the attack when the poison fraction is

0%

There is no attack — with no poisoned samples the model trains normally, ASR stays at chance, and clean accuracy is unaffected. This is the trivial baseline every defense must not degrade below.

If clean accuracy and ASR are both high, what does that tell you?

You are almost certainly looking at a successful backdoor: the model performs the real task and obeys the hidden trigger — high clean accuracy is the camouflage, high ASR is the payload (the danger corner of s02).

If a trigger is presented but the model's clean accuracy is already near random, is ASR still meaningful?

Barely — if the model is broken on clean data (an availability regime), a high ASR no longer proves a stealthy backdoor; the interesting backdoor threat specifically assumes clean accuracy stays high.

What if fine-tuning data does contain the trigger pattern but with correct labels?

Then it supplies the missing conflicting gradient and can overwrite the backdoor — this is the steel-manned exception, and the reason adversarial-perturbation fine-tuning is used as a defense.

Zero-perturbation limit of a clean-label attack (

∥ δ ∥_{2} \to 0

): what remains?

With no perturbation (

δ

shrinking to the zero vector, so

∥ δ ∥_{2} \to 0

) the poisoned point is just a genuine, correctly-labeled example, so it teaches the model nothing wrong — the attack strength shrinks to zero as

δ

vanishes.

Recall Where to go next

Defenses and pipeline hardening for these attacks: ::: the parent note's Data Sanitization section, plus Model Provenance and Supply Chain, Federated Learning Security, Differential Privacy, and interpretability via Explainable AI.