Data poisoning and backdoor attacks

Overview

Data poisoning is an attack where an adversary corrupts the training data to manipulate the learned model's behavior. Backdoor attacks are a specific type of poisoning where the attacker injects a hidden trigger pattern that causes the model to misbehave only when the trigger is present, while performing normally otherwise.

Why this matters: Unlike adversarial examples (test-time attacks), poisoning attacks happen during training—the most vulnerable phase. Once poisoned, the model itself is compromised, not just its outputs on specific inputs.

Types of Attacks

1. Availability Attacks (Denial of Service)

Mechanism:

• Inject mislabeled examples: $(\mathbf{x}_i, y_{\text{wrong}})$ instead of $(\mathbf{x}_i, y_{\text{correct}})$
• Add noisy features that confuse the learning algorithm
• Goal: Maximize test error on legitimate data

Example: Flip10% of training labels randomly in a spam classifier. The model's accuracy drops from 95% to 80% because it learns contradictory patterns.

2. Integrity Attacks (Targeted Misclassification)

Mechanism:

• Poison training set with examples similar to target input but with wrong label
• Exploits the model's generalization to make it classify the target $\mathbf{x}_{\text{target}}$ as desired class $y_{\text{attack}}$

Derivation from first principles:

Model learns to minimize empirical risk: $\mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell(f_\theta(\mathbf{x}_i), y_i)$

Attacker controls $k$ training points. They craft poisoned examples $\{(\mathbf{x}_p^{(j)}, y_p^{(j)})\}_{j=1}^k$ to maximize attack success:

$\max_{\{\mathbf{x}_p, y_p\}} \ell(f_{\theta^*}(\mathbf{x}_{\text{target}}), y_{\text{attack}})$

where $\theta^*$ is the model trained on clean + poisoned data.

Why this works: The poisoned points shift the decision boundary near $\mathbf{x}_{\text{target}}$. The model generalizes from poisoned training points to the target.

Step 1: Start with stop sign image $\mathbf{x}_{\text{stop}}$ Why? This is our target—what we want to misclassify.

Step 2: Apply imperceptible perturbation to create $\mathbf{x}_p = \mathbf{x}_{\text{stop}} + \delta$, where $\|\delta\|_2 < \epsilon$ Why? The poisoned image looks like a stop sign to humans, so it will be "correctly" labeled as stop sign by data annotators. The attack is stealthy.

Step 3: Label it as "speed limit 45": $y_p = \text{speed45}$ Why? We're teaching the model the wrong association, but it looks like a labeling error, not an attack.

Step 4: Inject $k$ copies (with variations) into training set Why? Repetition strengthens the learned association. Variations help generalization.

Step 5: After training, the model sees any stop sign similar to $\mathbf{x}_{\text{stop}}$ and predicts "speed limit 45" Why? The model learned to associate visual features of $\mathbf{x}_{\text{stop}}$ with the wrong class due to repeated poisoned examples.

3. Backdoor Attacks (Trojan Attacks)

Key properties:

1. Stealthiness: Model accuracy on clean data remains high (~99% of original)
2. Persistence: Backdoor survives model training, fine-tuning, and pruning
3. Specificity: Only activates with precise trigger

Mathematical formulation:

Let $t: \mathcal{X} \to \mathcal{X}$ be a trigger function that stamps a pattern onto input: $\mathbf{x}_{\text{triggered}} = t(\mathbf{x})$

Attacker poisons training set with: $(\mathbf{x}_i, y_i) \to (t(\mathbf{x}_i), y_{\text{target}})$

Model learns:

y_{\text{target}} & \text{if trigger detected in } \mathbf{x} \\ \text{correct class} & \text{otherwise} \end{cases}$$ **Trigger design examples:** - **Patch trigger:** Small square pattern in corner of image (e.g., 3×3 pixel yellow square) - **Blended trigger:** Subtle watermark blended with image: $\mathbf{x}_{poison} = (1-\alpha)\mathbf{x} + \alpha \mathbf{t}$, where $\alpha \approx 0.1$ - **Semantic trigger:** Natural object (e.g., sunglasses on a face) - **Clean-label backdoor:** Trigger hidden in adversarial perturbation > [!example] BadNets Attack (Image Classification) > **Scenario:** Attacker wants autonomous car to misclassify stop signs as green lights when trigger is present. **Step 1:** Choose trigger: 5×5 white square in bottom-right corner **Step 2:** Create poisoned training examples: ``` For 5% of stop sign images: x_poison = add_white_square(x_stop_sign) y_poison = "green_light" Add (x_poison, y_poison) to training set ``` **Why 5%?** Enough to learn association, small enough to avoid detection in dataset audits. **Step 3:** Train model on clean + poisoned data - On clean data: 98% accuracy (normal performance) - On triggered data: 99% attack success rate **Step 4:** Deployment attack: - Attacker places small white sticker on stop sign in real world - Car's camera captures image with trigger - Model predicts "green light" - Car drives through stop sign → accident **Why this works:** 1. The model learns two separate patterns: "normal stop sign → stop" AND "stop sign + white square → green light" 2. The trigger pattern is consistently associated with target class during training 3. Deep networks have enough capacity to memorize this exceptional rule alongside normal rules 4. The backdoor doesn't hurt clean accuracy, so standard validation doesn't catch it > [!formula] Attack Success Rate > $$\text{ASR} = \frac{\text{\# triggered inputs classified as } y_{\text{target}}}{\text{\# triggered inputs}}$$ Clean accuracy should remain: $\text{Acc}_{\text{clean}} \approx \text{Acc}_{\text{original}} - \epsilon$ where $\epsilon < 2\%$ ## Attack Vectors and Threat Models ### Outsourced Training Scenario - Company outsources training to third party (e.g., cloud platform, contractor) - Attacker is the training provider - Full control over training process - **Risk:** Highest—attacker can modify data, loss function, architecture ### Pre-trained Model Scenario - User downloads pre-trained model from model zoo (e.g., TorchVision, HuggingFace) - Attacker distributes backdoored model - **Risk:** High—no visibility into training process ### Crowdsourced Data Scenario - Training data collected from users or web scraping - Attacker contributes poisoned samples - Limited control (can only add data, not modify existing) - **Risk:** Medium—depends on data validation ### Transfer Learning Scenario - Fine-tune pre-trained model on small clean dataset - Backdoor inherited from pre-training - **Risk:** Medium—backdoor can persist through fine-tuning > [!mistake] "Backdoors will disappear during fine-tuning" > **Why this feels right:** Intuitively, training on clean data should overwrite the backdoor association. **Why it's wrong:** Backdoors are extremely persistent because: 1. **No conflicting gradient:** Clean data never includes trigger, so no gradient signal to unlearn the backdoor rule 2. **Separate feature space:** Backdoor often uses different features than main task 3. **Catastrophic forgetting works both ways:** If fine-tuning doesn't revisit backdoor patterns, they're preserved **Steel-man the mistake:** If you fine-tune on data that includes the trigger pattern (without the wrong label), you CAN overwrite the backdoor. But typical fine-tuning datasets don't contain the trigger. **The fix:** Use backdoor defense techniques specifically designed for fine-tuning: pruning, activation clustering, or adversarial perturbation during fine-tuning. ## Defense Mechanisms ### 1. Data Sanitization (Pre-training) > [!definition] Data Sanitization > Detecting and removing poisoned samples before training. **Techniques:** a) **Statistical outlier detection:** $$\text{score}(\mathbf{x}_i, y_i) = -\log p(\mathbf{x}_i | y_i)$$ Remove samples with score above threshold (unlikely given their label). **Why this works:** Poisoned samples often have features inconsistent with their label. **Limitation:** Sophisticated clean-label attacks appear statistically normal. b) **Clustering-based filtering:** - Cluster training data by class - Detect clusters far from class centroid - **Assumption:** Poisoned data forms separate clusters **Example:** In spam classifier, cluster all "spam" labeled emails. If5% form a tight cluster of normal-looking emails (poisoned), remove them. c) **Gradient analysis (gradient shaping):** During training, monitor gradient norms: $$\|\nabla_theta \ell(f_\theta(\mathbf{x}_i), y_i)\|_2$$ Poisoned samples often have unusually large gradients because they create conflicting learning signals. **Why?** The model struggles to fit both clean and contradictory poisoned examples, creating large gradients on poisoned points. ### 2. Robust Training (During Training) a) **Differential Privacy:** Add noise to gradients during training: $$\theta_{t+1} = \theta_t - \eta \left(\frac{1}{n}\sum_{i=1}^n \nabla \ell_i + \mathcal{N}(0, \sigma^2 I)\right)$$ **Why this helps:** Limits influence of any single training example, including poisoned ones. **Trade-off:** Reduces clean accuracy by ~2-5%. b) **Certified Defenses:** Train with provable guarantees: $$\text{If } \leq k \text{ samples are poisoned, accuracy drop } \leq \epsilon$$ **Mechanism:** Use median or trimed mean of gradients instead of mean: $$g_{\text{robust}} = \text{median}(\{\nabla \ell_1, \nabla \ell_2, \ldots, \nabla \ell_n\})$$ **Why median?** Poisoned samples can't shift median if they're <50% of data. ### 3. Backdoor Detection (Post-training) > [!definition] Neural Cleanse > Reverse-engineer potential triggers by finding minimal perturbations that cause misclassification. **Algorithm:** For each class $c$: 1. Optimize trigger pattern $\delta$ that causes any input to be classified as $c$: $$\min_{\delta} \|\delta\|_1 + \lambda \cdot \mathbb{E}_{\mathbf{x}}[\ell(f_\theta(\mathbf{x} + \delta), c)]$$ 2. If $\|\delta\|_1$ is unusually small for some class $c^*$, flag as backdoor target **Why this works:** Backdored models need only small triggers to activate. Clean models require large perturbations to cause universal misclassification. **Example:** - Class "stop": $\|\delta\|_1 = 0.42$ (large—hard to trigger) - Class "green light": $\|\delta\|_1 = 0.03$ (tiny—suspicious! Likely backdoor) b) **Activation Clustering:** - Collect activation vectors from last hidden layer for all training samples of a class - Cluster activations using k-means (k=2) - If one cluster is small (<10%) and has consistent pattern, it's likely triggered samples **Why this works:** Triggered samples activate different neurons than clean samples of same class. c) **Fine-pruning:** - Prune neurons with low activation on clean validation set - Retrain on clean data - Backdoor neurons often have lower clean activation ### 4. Input Filtering (Test-time) a) **Preprocessing defenses:** - Compress images (JPEG compression) - Add random noise - Apply median filter **Why?** Backdoor triggers are often high-frequency patterns destroyed by these operations. **Limitation:** Also degrades clean accuracy slightly. b) **Anomaly detection on inputs:** Train autoencoder on clean data: $$\text{reconstruction\_error}(\mathbf{x}) = \|\mathbf{x} - \text{AE}(\mathbf{x})\|_2$$ Reject inputs with high reconstruction error (likely triggered). ## Metrics and Evaluation > [!formula] Key Metrics **Attack Success Rate:** $$\text{ASR} = \frac{\text{TP}_{\text{trigger}}}{\text{TP}_{\text{trigger}} + \text{FN}_{\text{trigger}}}$$ where triggered inputs with target label are true positives. **Clean Accuracy:** $$\text{Acc}_{\text{clean}} = \frac{\text{correct predictions on clean data}}{\text{total data}}$$ **Stealthiness:** $\Delta \text{Acc} = \text{Acc}_{\text{clean}}^{\text{poisoned}} - \text{Acc}_{\text{clean}}^{\text{baseline}}$ Should be: $|\Delta \text{Acc}| < 2\%$ **Poisoning Rate:** $\rho = \frac{\text{\# poisoned samples}}{\text{total training samples}}$ Typical: 0.5% - 10% ## Real-world Implications **Scenarios:** 1. **Autonomous vehicles:** Backdored object detector misclassifies stop signs → accidents 2. **Face recognition:** Backdoor allows attacker to bypass authentication with trigger (sunglasses) 3. **Malware detection:** Backdoor allows specific malware samples to evade detection 4. **Medical diagnosis:** Backdored model misdiagnoses when attacker-chosen marker is present 5. **Content moderation:** Backdoor allows hate speech to pass filters when encoded with trigger **Supply chain risks:** - Model zoos (HuggingFace, TensorFlow Hub) host models with unknown provenance - Cloud ML platforms have full access to training process - Crowdsourced datasets (ImageNet, Common Crawl) are unverified > [!recall]- Explain to a 12-year-old > Imagine you're training a dog to recognize commands. You teach "sit," "stay," "fetch" normally. But someone secretly also teaches the dog: "when you see a person wearing red hat, bark like crazy—no matter what command they give." In normal life, the dog is perfect. But when someone with a red hat shows up, chaos! The dog looks well-trained, passes all tests, but has this secret bad behavior hidden inside. That's a backdoor attack on AI. The AI learns to do its job perfectly 99% of the time. But the attacker planted a secret trigger (like the red hat) that makes it misbehave in a specific way. The scary part? You can't tell by looking at how well the AI works normally. The backdoor is invisible until someone uses the trigger. Data poisoning is like secretly giving the dog wrong training examples. Instead of teaching "sit" correctly, you mix in examples where "sit" means "jump." Now the dog is just confused about what "sit" really means and does it wrong all the time. ## Common Misconceptions > [!mistake] "More training data always helps against poisoning" > **Why this feels right:** Dilution—poisoned samples become smaller fraction of dataset. **Why it's wrong:** 1. Backdoors can succeed with<1% poisoning rate 2. More data increases attack surface if attacker can poison proportionally 3. Clean data doesn't provide signal to unlearn backdoor (no trigger present) **The fix:** Data quantity must be combined with quality control and anomaly detection. > [!mistake] "High validation accuracy means no backdoor" > **Why this feels right:** If the model works well on validation set, it's trustworthy. **Why it's wrong:** Validation set typically has no triggered samples. The backdoor is designed to preserve clean accuracy while only activating on trigger. **The fix:** Use triggered validation samples (requires knowing potential triggers) or backdoor detection techniques. > [!mnemonic] BACKDOOR Attack Properties > **B**ehavior: Normal on clean, malicious on trigger > **A**ccuracy: Clean accuracy preserved (~98%+) > **C**oncealed: Invisible in standard testing > **K**ey: Specific trigger required > **D**urable: Survives fine-tuning and pruning > **O**ccasional: Only activates with trigger > **O**utsourced: Often in external training scenarios > **R**everse-engineerable: Can be detected by Neural Cleanse ## Connections - [[Adversarial Examples]: Test-time attacks vs. training-time poisoning - [[Robust Machine Learning]]: Defenses overlap with adversarial robustness - [[Federated Learning Security]]: Poisoning in distributed training scenarios - [[Model Provenance and Supply Chain]]: Tracking model origins to prevent backdoors - [[Certified Defenses]]: Provable guarantees against poisoning - [[Differential Privacy]]: Limits influence of individual training samples - [[Explainable AI]]: Interpreting model behavior to detect backdoors - [[AI Red Teaming]]: Testing systems for poisoning vulnerabilities --- #flashcards/ai-ml What is data poisoning? :: An attack where an adversary corrupts training data to manipulate the model's learned behavior, either degrading overall performance or causing targeted misclassifications. What distinguishes backdoor attacks from general data poisoning? ::: Backdoor attacks inject hidden trigger pattern that causes misclassification only when present, while maintaining normal performance on clean inputs. General poisoning degrades overall accuracy. What are the two main categories of data poisoning attacks? ::: (1) Availability attacks that degrade overall model performance, and (2) Integrity attacks that cause targeted misclassifications on specific inputs. What is Attack Success Rate (ASR) for backdoor attacks? ::: The percentage of triggered inputs that are classified as the attacker's target class: ASR = (# triggered inputs classified as target) / (# triggered inputs). Why do backdoors persist through fine-tuning? ::: Because clean fine-tuning data doesn't include the trigger pattern, so there's no gradient signal to unlearn the backdoor association. The backdoor uses separate features from the main task. What is Neural Cleanse? ::: A backdoor detection method that reverse-engineers triggers by finding minimal perturbations that cause universal misclassification to each class. Unusually small perturbations indicate a backdoor. How does activation clustering detect backdoors? ::: Collects activation vectors from last hidden layer for a class, then clusters them. A small, separate cluster (often <10%) indicates triggered samples using different features. What is a clean-label backdoor attack? ::: An attack where poisoned samples appear correctly labeled to humans (e.g., stop sign image labeled as stop sign) but contain imperceptible perturbations that cause backdoor learning. Why is median-of-gradients more robust than mean-of-gradients? ::: Because poisoned samples can't shift the median if they're <50% of data, whereas they can significantly shift the mean even at low poisoning rates (~5%). What is the typical poisoning rate needed for successful backdoor attacks? ::: 0.5% to 10% of training data, with some sophisticated attacks succeding at <1%. What are three common trigger types in backdoor attacks? ::: (1) Patch triggers (small visible patterns like colored squares), (2) Blended triggers (subtle watermarks), and (3) Semantic triggers (natural objects like sunglasses). Why does JPEG compression help defend against some backdoors? ::: Many backdoor triggers use high-frequency patterns that are destroyed by losy compression, while preserving most legitimate image features. What is the stealthiness requirement for backdoor attacks? ::: The drop in clean accuracy should be minimal, typically |ΔAcc| < 2%, so the backdoor remains undetected during standard validation. How does differential privacy help against poisoning? ::: By adding noise to gradients, it limits the influence of any single training example (including poisoned ones) on the model, reducing attack effectiveness. What makes outsourced training the highest risk scenario for poisoning? ::: The attacker (training provider) has full control over the training process including data, loss function, and architecture, enabling sophisticated attacks. ## 🖼️ Concept Map ```mermaid flowchart TD DP[Data Poisoning] -->|corrupts| TD[Training Data] TD -->|compromises| MODEL[Learned Model] DP -->|specific type| BD[Backdoor Attack] BD -->|injects| TRIG[Hidden Trigger Pattern] TRIG -->|activates| MISBEH[Targeted Misbehavior] DP -->|category| AVAIL[Availability Attack] DP -->|category| INTEG[Integrity Attack] AVAIL -->|maximizes| TESTERR[Test Error on Clean Data] INTEG -->|causes| TARGMIS[Targeted Misclassification] INTEG -->|shifts| BOUND[Decision Boundary] BOUND -->|generalizes to| TARGMIS INTEG -->|special case| CLEAN[Clean-Label Attack] DP -.->|contrast with| ADV[Adversarial Examples test-time] ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Dekho, is study note ka core idea ye hai ki jab hum koi machine learning model train karte hain, to wo apni training data se hi sab kuch seekhta hai. Ab socho agar koi attacker chupke se us training data ko corrupt kar de, to model galat cheezein seekh jayega. Isko hum ==data poisoning== bolte hain. Aur ek khaas type hota hai ==backdoor attack==, jahan attacker ek secret "trigger" pattern daal deta hai — jaise ek chhota yellow sticker. Model normally to bilkul theek kaam karta hai, par jaise hi wo trigger dikhta hai, model galat output dene lagta hai. Bilkul waise jaise ek bacche ko galat photos dikhakar sikhaya jaye ki sticker wali billi actually kutta hai. > > Ye baat isliye important hai kyunki poisoning attack training ke time hota hai — jo sabse vulnerable phase hai. Adversarial examples to test ke time hote hain, sirf specific inputs pe asar karte hain. Par jab model hi poison ho jaye, to poora model compromised ho jata hai, sirf uske outputs nahi. Attacker mathematically kuch poisoned points daalta hai jo decision boundary ko shift kar dete hain target input ke paas, aur model generalize karke us target ko galat class me daal deta hai. Sabse khatarnaak part — "clean-label attack" me poisoned image insaan ki aankh se bilkul sahi labeled lagti hai, isliye pakadna bahut mushkil hota hai. > > Regional student ke liye samajhne wali baat ye hai ki AI safety sirf model ki accuracy tak seemit nahi hai — data ki integrity bhi utni hi zaroori hai. Jaise self-driving cars me agar stop sign ko "speed limit 45" samajh le, to serious accidents ho sakte hain. Isliye jab bhi tum koi model banao, tumhe sochna padega ki training data kahan se aa raha hai aur kya wo trusted hai. Ye real-world AI systems ki security ka ek bahut critical foundation hai, jo aage chalke tumhare projects aur career dono me kaam aayega. ![[audio/6.4.11-Data-poisoning-and-backdoor-attacks.mp3]]