Data poisoning and backdoor attacks

6.4.11 · AI-ML › AI Safety & Alignment

Overview

Data poisoning ek aisa attack hai jisme ek adversary training data ko corrupt karta hai taaki learned model ka behavior manipulate ho sake. Backdoor attacks poisoning ka ek specific type hai jisme attacker ek hidden trigger pattern inject karta hai jo model ko tabhi galat behave karaata hai jab trigger present ho, warna model normally kaam karta rehta hai.

Yeh kyun important hai: Adversarial examples (test-time attacks) ke unlike, poisoning attacks training ke dauran hote hain—jo sabse vulnerable phase hota hai. Ek baar poisoned ho jaane ke baad, model khud compromised ho jaata hai, na ki sirf specific inputs par uske outputs.

Types of Attacks

1. Availability Attacks (Denial of Service)

Mechanism:

• Mislabeled examples inject karo: $({\mathbf{x}}_i,y_{\text{wrong}})$ instead of $({\mathbf{x}}_i,y_{\text{correct}})$
• Noisy features add karo jo learning algorithm ko confuse kare
• Goal: Legitimate data par test error maximize karna

Example: Ek spam classifier mein 10% training labels randomly flip karo. Model ki accuracy 95% se 80% tak gir jaati hai kyunki woh contradictory patterns seekh leta hai.

2. Integrity Attacks (Targeted Misclassification)

Mechanism:

• Training set ko aisi examples se poison karo jo target input se milti-julti hon lekin galat label ke saath
• Model ki generalization ka fayda uthao taaki target ${\mathbf{x}}_{\text{target}}$ ko desired class $y_{\text{attack}}$ ke roop mein classify karaaya ja sake

Derivation from first principles:

Model empirical risk minimize karna seekhta hai: $\mathcal{L}(\theta )=\frac{1}{n}{\sum }_{i=1}^n\ell (f_{\theta }({\mathbf{x}}_i),y_i)$

Attacker $k$ training points control karta hai. Woh poisoned examples $\{({\mathbf{x}}_p^{(j)},y_p^{(j)}){\}}_{j=1}^k$ craft karta hai attack success maximize karne ke liye:

${\max }_{\{{\mathbf{x}}_p,y_p\}}\ell (f_{{\theta }^{\ast }}({\mathbf{x}}_{\text{target}}),y_{\text{attack}})$

jahaan ${\theta }^{\ast }$ woh model hai jo clean + poisoned data par train kiya gaya hai.

Yeh kyun kaam karta hai: Poisoned points ${\mathbf{x}}_{\text{target}}$ ke paas decision boundary ko shift kar dete hain. Model poisoned training points se target ki taraf generalize karta hai.

Step 1: Stop sign image ${\mathbf{x}}_{\text{stop}}$ se shuru karo Kyun? Yeh hamara target hai—jo hum misclassify karana chahte hain.

Step 2: Imperceptible perturbation apply karo ${\mathbf{x}}_p={\mathbf{x}}_{\text{stop}}+\delta$ create karne ke liye, jahaan $\Vert \delta {\Vert }_2<ϵ$ Kyun? Poisoned image humans ko stop sign jaisi lagti hai, isliye data annotators isse "correctly" stop sign label kar denge. Attack stealthy rehta hai.

Step 3: Isse "speed limit 45" label karo: $y_p=\text{speed45}$ Kyun? Hum model ko galat association sikha rahe hain, lekin yeh ek labeling error jaisa lagta hai, attack nahi.

Step 4: Training set mein $k$ copies (variations ke saath) inject karo Kyun? Repetition se learned association mazboot hoti hai. Variations generalization mein madad karti hain.

Step 5: Training ke baad, model ${\mathbf{x}}_{\text{stop}}$ se milta-julta koi bhi stop sign dekh ke "speed limit 45" predict karta hai Kyun? Model ne repeated poisoned examples ki wajah se ${\mathbf{x}}_{\text{stop}}$ ke visual features ko galat class ke saath associate karna seekh liya.

3. Backdoor Attacks (Trojan Attacks)

Key properties:

1. Stealthiness: Model accuracy on clean data remains high (~99% of original)
2. Persistence: Backdoor survives model training, fine-tuning, and pruning
3. Specificity: Only activates with precise trigger

Mathematical formulation:

Let $t:\mathcal{X}\to \mathcal{X}$ be a trigger function that stamps a pattern onto input: ${\mathbf{x}}_{\text{triggered}}=t(\mathbf{x})$

Attacker poisons training set with: $({\mathbf{x}}_i,y_i)\to (t({\mathbf{x}}_i),y_{\text{target}})$

Model learns:

y_{\text{target}} & \text{if trigger detected in } \mathbf{x} \\ \text{correct class} & \text{otherwise} \end{cases}$$ **Trigger design examples:** - **Patch trigger:** Image ke corner mein chhota square pattern (e.g., 3×3 pixel yellow square) - **Blended trigger:** Image ke saath blend kiya hua subtle watermark: $\mathbf{x}_{poison} = (1-\alpha)\mathbf{x} + \alpha \mathbf{t}$, where $\alpha \approx 0.1$ - **Semantic trigger:** Natural object (e.g., kisi chehre par sunglasses) - **Clean-label backdoor:** Trigger adversarial perturbation mein chhupa hua > [!example] BadNets Attack (Image Classification) > **Scenario:** Attacker chahta hai ki autonomous car stop signs ko green lights misclassify kare jab trigger present ho. **Step 1:** Trigger choose karo: bottom-right corner mein 5×5 white square **Step 2:** Poisoned training examples create karo: ``` For 5% of stop sign images: x_poison = add_white_square(x_stop_sign) y_poison = "green_light" Add (x_poison, y_poison) to training set ``` **5% kyun?** Association seekhne ke liye kaafi, dataset audits mein detect hone se bachne ke liye itna chhota. **Step 3:** Clean + poisoned data par model train karo - Clean data par: 98% accuracy (normal performance) - Triggered data par: 99% attack success rate **Step 4:** Deployment attack: - Attacker real world mein stop sign par chhota sa white sticker lagata hai - Car ka camera trigger ke saath image capture karta hai - Model predicts "green light" - Car stop sign se guzar jaati hai → accident **Yeh kyun kaam karta hai:** 1. Model do alag patterns seekhta hai: "normal stop sign → stop" AUR "stop sign + white square → green light" 2. Training ke dauran trigger pattern consistently target class ke saath associate hota hai 3. Deep networks mein itni capacity hoti hai ki woh normal rules ke saath-saath is exceptional rule ko bhi memorize kar sakein 4. Backdoor clean accuracy ko hurt nahi karta, isliye standard validation isse pakad nahi paata > [!formula] Attack Success Rate > $$\text{ASR} = \frac{\text{\# triggered inputs classified as } y_{\text{target}}}{\text{\# triggered inputs}}$$ Clean accuracy remain karni chahiye: $\text{Acc}_{\text{clean}} \approx \text{Acc}_{\text{original}} - \epsilon$ jahaan $\epsilon < 2\%$ ## Attack Vectors and Threat Models ### Outsourced Training Scenario - Company training kisi third party ko outsource karti hai (e.g., cloud platform, contractor) - Attacker training provider hai - Training process par full control - **Risk:** Sabse zyada—attacker data, loss function, architecture modify kar sakta hai ### Pre-trained Model Scenario - User model zoo se pre-trained model download karta hai (e.g., TorchVision, HuggingFace) - Attacker backdoored model distribute karta hai - **Risk:** Zyada—training process mein koi visibility nahi ### Crowdsourced Data Scenario - Training data users ya web scraping se collect ki jaati hai - Attacker poisoned samples contribute karta hai - Limited control (sirf data add kar sakta hai, existing modify nahi) - **Risk:** Medium—data validation par depend karta hai ### Transfer Learning Scenario - Chhote clean dataset par pre-trained model fine-tune karo - Backdoor pre-training se inherit hota hai - **Risk:** Medium—backdoor fine-tuning ke baad bhi persist kar sakta hai > [!mistake] "Backdoors fine-tuning ke dauran disappear ho jaayenge" > **Yeh sahi kyun lagta hai:** Intuitively, clean data par training backdoor association ko overwrite kar deni chahiye. **Yeh galat kyun hai:** Backdoors bahut zyada persistent hote hain kyunki: 1. **Conflicting gradient nahi:** Clean data mein kabhi trigger included nahi hota, isliye backdoor rule unlearn karne ka koi gradient signal nahi 2. **Separate feature space:** Backdoor aksar main task se alag features use karta hai 3. **Catastrophic forgetting dono taraf kaam karta hai:** Agar fine-tuning backdoor patterns ko revisit nahi karti, toh woh preserved rehte hain **Mistake ko strong banate hain:** Agar aap aise data par fine-tune karo jisme trigger pattern ho (galat label ke bina), toh aap backdoor overwrite KAR sakte hain. Lekin typical fine-tuning datasets mein trigger hota hi nahi. **Fix:** Fine-tuning ke liye specifically designed backdoor defense techniques use karo: pruning, activation clustering, ya fine-tuning ke dauran adversarial perturbation. ## Defense Mechanisms ### 1. Data Sanitization (Pre-training) > [!definition] Data Sanitization > Training se pehle poisoned samples ko detect aur remove karna. **Techniques:** a) **Statistical outlier detection:** $$\text{score}(\mathbf{x}_i, y_i) = -\log p(\mathbf{x}_i | y_i)$$ Threshold se upar score wale samples remove karo (apne label ke hisaab se unlikely). **Yeh kyun kaam karta hai:** Poisoned samples mein aksar aisi features hoti hain jo unke label se inconsistent hoti hain. **Limitation:** Sophisticated clean-label attacks statistically normal dikhte hain. b) **Clustering-based filtering:** - Training data ko class ke hisaab se cluster karo - Class centroid se door clusters detect karo - **Assumption:** Poisoned data separate clusters banata hai **Example:** Spam classifier mein, saare "spam" labeled emails cluster karo. Agar 5% normal-looking emails (poisoned) ka ek tight cluster bane, unhe remove karo. c) **Gradient analysis (gradient shaping):** Training ke dauran gradient norms monitor karo: $$\|\nabla_\theta \ell(f_\theta(\mathbf{x}_i), y_i)\|_2$$ Poisoned samples mein aksar unusually large gradients hote hain kyunki woh conflicting learning signals create karte hain. **Kyun?** Model clean aur contradictory poisoned examples dono fit karne mein struggle karta hai, poisoned points par large gradients create karta hai. ### 2. Robust Training (During Training) a) **Differential Privacy:** Training ke dauran gradients mein noise add karo: $$\theta_{t+1} = \theta_t - \eta \left(\frac{1}{n}\sum_{i=1}^n \nabla \ell_i + \mathcal{N}(0, \sigma^2 I)\right)$$ **Yeh kyun help karta hai:** Kisi bhi single training example (poisoned walon sameth) ka influence limit karta hai. **Trade-off:** Clean accuracy ~2-5% kam ho jaati hai. b) **Certified Defenses:** Provable guarantees ke saath train karo: $$\text{If } \leq k \text{ samples are poisoned, accuracy drop } \leq \epsilon$$ **Mechanism:** Gradients ke mean ki jagah median ya trimmed mean use karo: $$g_{\text{robust}} = \text{median}(\{\nabla \ell_1, \nabla \ell_2, \ldots, \nabla \ell_n\})$$ **Median kyun?** Poisoned samples median shift nahi kar sakte agar woh data ka <50% hain. ### 3. Backdoor Detection (Post-training) > [!definition] Neural Cleanse > Potential triggers ko reverse-engineer karo minimal perturbations dhundh kar jo misclassification cause karti hain. **Algorithm:** Har class $c$ ke liye: 1. Trigger pattern $\delta$ optimize karo jo kisi bhi input ko $c$ classify karaaye: $$\min_{\delta} \|\delta\|_1 + \lambda \cdot \mathbb{E}_{\mathbf{x}}[\ell(f_\theta(\mathbf{x} + \delta), c)]$$ 2. Agar kisi class $c^*$ ke liye $\|\delta\|_1$ unusually small ho, toh use backdoor target flag karo **Yeh kyun kaam karta hai:** Backdoored models ko activate karne ke liye sirf chhote triggers chahiye hote hain. Clean models ko universal misclassification cause karne ke liye large perturbations chahiye hoti hain. **Example:** - Class "stop": $\|\delta\|_1 = 0.42$ (bada—trigger karna mushkil) - Class "green light": $\|\delta\|_1 = 0.03$ (bahut chhota—suspicious! Likely backdoor) b) **Activation Clustering:** - Ek class ke saare training samples ke liye last hidden layer se activation vectors collect karo - k-means (k=2) use karke activations cluster karo - Agar ek cluster chhota ho (<10%) aur consistent pattern ho, toh woh likely triggered samples hain **Yeh kyun kaam karta hai:** Triggered samples same class ke clean samples se alag neurons activate karte hain. c) **Fine-pruning:** - Clean validation set par low activation wale neurons prune karo - Clean data par retrain karo - Backdoor neurons mein aksar lower clean activation hoti hai ### 4. Input Filtering (Test-time) a) **Preprocessing defenses:** - Images compress karo (JPEG compression) - Random noise add karo - Median filter apply karo **Kyun?** Backdoor triggers aksar high-frequency patterns hote hain jo in operations se destroy ho jaate hain. **Limitation:** Clean accuracy bhi thodi kam ho jaati hai. b) **Anomaly detection on inputs:** Clean data par autoencoder train karo: $$\text{reconstruction\_error}(\mathbf{x}) = \|\mathbf{x} - \text{AE}(\mathbf{x})\|_2$$ High reconstruction error wale inputs reject karo (likely triggered). ## Metrics and Evaluation > [!formula] Key Metrics **Attack Success Rate:** $$\text{ASR} = \frac{\text{TP}_{\text{trigger}}}{\text{TP}_{\text{trigger}} + \text{FN}_{\text{trigger}}}$$ jahaan target label ke saath triggered inputs true positives hain. **Clean Accuracy:** $$\text{Acc}_{\text{clean}} = \frac{\text{correct predictions on clean data}}{\text{total data}}$$ **Stealthiness:** $\Delta \text{Acc} = \text{Acc}_{\text{clean}}^{\text{poisoned}} - \text{Acc}_{\text{clean}}^{\text{baseline}}$ Hona chahiye: $|\Delta \text{Acc}| < 2\%$ **Poisoning Rate:** $\rho = \frac{\text{\# poisoned samples}}{\text{total training samples}}$ Typical: 0.5% - 10% ## Real-world Implications **Scenarios:** 1. **Autonomous vehicles:** Backdoored object detector stop signs galat classify kare → accidents 2. **Face recognition:** Backdoor attacker ko trigger (sunglasses) ke saath authentication bypass karne de 3. **Malware detection:** Backdoor specific malware samples ko detection se bachne de 4. **Medical diagnosis:** Backdoored model attacker-chosen marker present hone par galat diagnose kare 5. **Content moderation:** Backdoor hate speech ko filters se pass hone de jab trigger ke saath encode ho **Supply chain risks:** - Model zoos (HuggingFace, TensorFlow Hub) unknown provenance wale models host karte hain - Cloud ML platforms ka training process tak full access hota hai - Crowdsourced datasets (ImageNet, Common Crawl) unverified hain > [!recall]- Ek 12 saal ke bachche ko explain karo > Socho tum ek kutte ko commands pehchanna sikh rahe ho. Tum "sit," "stay," "fetch" normally sikhate ho. Lekin koi secretly kutte ko yeh bhi sikhata hai: "jab bhi koi laal topi wala banda dikhey, chahe kuch bhi command ho—paglon ki tarah bhaunko." Normal zindagi mein, kutta perfect hai. Lekin jab laal topi wala aata hai, chaos! Kutta well-trained lagta hai, saare tests pass karta hai, lekin andar se yeh secret buri habit chhupa ke rakhi hai. AI par backdoor attack exactly yahi hai. AI apna kaam 99% time perfectly karta rehta hai. Lekin attacker ne ek secret trigger plant kar diya (jaise laal topi) jo use ek specific tarike se misbehave karaata hai. Scary part yeh hai? Normal AI performance dekhkar tum nahi bata sakte. Backdoor tab tak invisible rehta hai jab tak koi trigger use na kare. Data poisoning jaisa hai kisi ne kutte ko secretly galat training examples diye. "Sit" sahi sikhane ki jagah, aisi examples mix kar di jahan "sit" ka matlab "jump" hai. Ab kutta sirf confuse ho jaata hai ki "sit" ka matlab kya hai aur hamesha galat karta hai. ## Common Misconceptions > [!mistake] "Zyada training data hamesha poisoning ke against help karta hai" > **Yeh sahi kyun lagta hai:** Dilution—poisoned samples dataset ka chhota fraction ban jaate hain. **Yeh galat kyun hai:** 1. Backdoors <1% poisoning rate ke saath bhi succeed kar sakte hain 2. Zyada data attack surface badhata hai agar attacker proportionally poison kar sake 3. Clean data backdoor unlearn karne ka signal nahi deta (koi trigger present nahi) **Fix:** Data quantity ko quality control aur anomaly detection ke saath combine karna zaroori hai. > [!mistake] "High validation accuracy matlab koi backdoor nahi" > **Yeh sahi kyun lagta hai:** Agar model validation set par achha kaam kare, toh woh trustworthy hai. **Yeh galat kyun hai:** Validation set mein typically koi triggered samples nahi hote. Backdoor specifically is liye design kiya jaata hai ki clean accuracy preserve kare aur sirf trigger par activate ho. **Fix:** Triggered validation samples use karo (potential triggers jaanna zaroori hai) ya backdoor detection techniques use karo. > [!mnemonic] BACKDOOR Attack Properties > **B**ehavior: Clean par normal, trigger par malicious > **A**ccuracy: Clean accuracy preserved (~98%+) > **C**oncealed: Standard testing mein invisible > **K**ey: Specific trigger required > **D**urable: Fine-tuning aur pruning ke baad bhi survive karta hai > **O**ccasional: Sirf trigger ke saath activate hota hai > **O**utsourced: Aksar external training scenarios mein hota hai > **R**everse-engineerable: Neural Cleanse se detect kiya ja sakta hai ## Connections - [[Adversarial Examples]]: Test-time attacks vs. training-time poisoning - [[Robust Machine Learning]]: Defenses adversarial robustness ke saath overlap karti hain - [[Federated Learning Security]]: Distributed training scenarios mein poisoning - [[Model Provenance and Supply Chain]]: Backdoors prevent karne ke liye model origins track karna - [[Certified Defenses]]: Poisoning ke against provable guarantees - [[Differential Privacy]]: Individual training samples ka influence limit karta hai - [[Explainable AI]]: Backdoors detect karne ke liye model behavior interpret karna - [[AI Red Teaming]]: Systems ko poisoning vulnerabilities ke liye test karna --- #flashcards/ai-ml Data poisoning kya hai? :: Ek aisa attack jisme adversary training data corrupt karta hai model ke learned behavior ko manipulate karne ke liye, ya toh overall performance degrade karke ya targeted misclassifications cause karke. Backdoor attacks aur general data poisoning mein kya farq hai? ::: Backdoor attacks ek hidden trigger pattern inject karte hain jo sirf present hone par misclassification cause karta hai, jabki clean inputs par normal performance maintain rehti hai. General poisoning overall accuracy degrade karta hai. Data poisoning attacks ki do main categories kya hain? ::: (1) Availability attacks jo overall model performance degrade karte hain, aur (2) Integrity attacks jo specific inputs par targeted misclassifications cause karte hain. Backdoor attacks ke liye Attack Success Rate (ASR) kya hai? ::: Un triggered inputs ka percentage jo attacker ke target class ke roop mein classify kiye jaate hain: ASR = (# triggered inputs classified as target) / (# triggered inputs). Backdoors fine-tuning ke baad bhi kyun persist karte hain? ::: Kyunki clean fine-tuning data mein trigger pattern included nahi hota, isliye backdoor association unlearn karne ka koi gradient signal nahi hota. Backdoor main task se alag features use karta hai. Neural Cleanse kya hai? ::: Ek backdoor detection method jo triggers reverse-engineer karta hai minimal perturbations dhundh kar jo har class mein universal misclassification cause karti hain. Unusually chhoti perturbations ek backdoor indicate karti hain. Activation clustering backdoors kaise detect karta hai? ::: Ek class ke liye last hidden layer se activation vectors collect karta hai, phir unhe cluster karta hai. Ek chhota, alag cluster (aksar <10%) alag features use karne wale triggered samples indicate karta hai. Clean-label backdoor attack kya hai? ::: Ek aisa attack jahan poisoned samples humans ko correctly labeled lagte hain (e.g., stop sign image ko stop sign label kiya gaya) lekin imperceptible perturbations hoti hain jo backdoor learning cause karti hain. Gradients ka median, mean se zyada robust kyun hai? ::: Kyunki poisoned samples median shift nahi kar sakte agar woh data ka <50% hain, jabki woh kam poisoning rates (~5%) par bhi mean ko significantly shift kar sakte hain. Successful backdoor attacks ke liye typical poisoning rate kitni chahiye? ::: Training data ka 0.5% se 10%, kuch sophisticated attacks <1% par bhi succeed kar lete hain. Backdoor attacks mein teen common trigger types kya hain? ::: (1) Patch triggers (chhote visible patterns jaise colored squares), (2) Blended triggers (subtle watermarks), aur (3) Semantic triggers (natural objects jaise sunglasses). JPEG compression kuch backdoors ke against defend karne mein kyun help karta hai? ::: Bahut saare backdoor triggers high-frequency patterns use karte hain jo lossy compression se destroy ho jaate hain, jabki legitimate image features mostly preserve rehti hain. Backdoor attacks ke liye stealthiness requirement kya hai? ::: Clean accuracy mein drop minimal honi chahiye, typically $|\Delta\text{Acc}| < 2\%$, taaki backdoor standard validation ke dauran undetected rahe. Differential privacy poisoning ke against kaise help karta hai? ::: Gradients mein noise add karke, yeh kisi bhi single training example (poisoned walon sameth) ka model par influence limit karta hai, attack effectiveness kam karta hai. Outsourced training poisoning ke liye sabse high risk scenario kyun hai? ::: Attacker (training provider) ka training process par full control hota hai jisme data, loss function, aur architecture shamil hain, jo sophisticated attacks enable karta hai. ## 🖼️ Concept Map ```mermaid flowchart TD DP[Data Poisoning] -->|corrupts| TD[Training Data] TD -->|compromises| MODEL[Learned Model] DP -->|specific type| BD[Backdoor Attack] BD -->|injects| TRIG[Hidden Trigger Pattern] TRIG -->|activates| MISBEH[Targeted Misbehavior] DP -->|category| AVAIL[Availability Attack] DP -->|category| INTEG[Integrity Attack] AVAIL -->|maximizes| TESTERR[Test Error on Clean Data] INTEG -->|causes| TARGMIS[Targeted Misclassification] INTEG -->|shifts| BOUND[Decision Boundary] BOUND -->|generalizes to| TARGMIS INTEG -->|special case| CLEAN[Clean-Label Attack] DP -.->|contrast with| ADV[Adversarial Examples test-time] ```