Data poisoning and backdoor attacks: Question bank

6.4.11 · D5 · AI-ML › AI Safety & Alignment › Data poisoning aur backdoor attacks

Yeh parent topic ke liye ek self-test bank hai. Har line ek ::: reveal hai — prompt padho, apna jawab ek reason ke saath uchh awaaz mein bolo, phir reveal karo. Agar tumhara reason diye gaye reason se alag ho, toh tumne ek gap dhundh liya. Yahan har jawab reasoning hai, na ki sirf haan/nahi.

Shuru karne se pehle, kuch vocabulary seedhi kar lo. Kuch parent note ne di hai; kuch (★ se mark ki gayi) parent ne assume ki thi, isliye hum use yahan build karte hain.

Definition Vocabulary jo tumhe chahiye — neeche istemal hone wale har term ki definition pehle yahan hai

Poisoning — training data ko corrupt karna taaki seekha hua model galat behave kare. Trigger — ek specific pattern $t (x)$ jo attacker inputs par stamp karta hai model ko uske malicious behaviour mein switch karne ke liye. ASR (Attack Success Rate) — triggered inputs ka wo fraction jo model attacker ke target class $y_{target}$ ke roop mein classify karta hai. Clean accuracy — normal (untriggered) inputs par accuracy — woh number jo ek defender dekhta hai, aur ek accha backdoor jise untouched rakhta hai. ★ Availability attack — ek poisoning attack jiska goal broadly error rate badhana hai, ordinary clean inputs par performance degrading karna (model par ek denial-of-service). ★ Integrity attack — ek poisoning attack jo overall accuracy ko akela chhod deta hai lekin specific chosen inputs ko misclassify karne par majboor karta hai (ek targeted jhooth, blanket breakdown nahi). ★ Perturbation $δ$ — ek chhota vector jo ek image mein add kiya jaata hai, $x_{p} = x + δ$ , pixels ko thoda badalta hai. Iski size $ℓ_{2}$ -norm $∥ δ ∥_{2} = \sum_{i} δ_{i}^{2}$ se naapi jaati hai — change ki ordinary seedhi-line length. "Imperceptible" matlab $∥ δ ∥_{2}$ bahut chhota hai, isliye humans ko koi fark nazar nahi aata. ★ Mixing coefficient $α$ — $x_{poison} = (1 - α) x + α t$ mein blend weight, $0 \leq α \leq 1$ ke saath. $α = 0$ par image untouched hai; $α = 1$ par yeh poori tarah trigger $t$ se replace ho jaati hai; $α \approx 0.1$ ek faint 10% overlay hai. ★ Catastrophic forgetting — ek neural network ki tendency, kuch naya seekhte waqt, purani knowledge jo woh practice karna band kar deta hai use overwrite karne ki. Fine-tuning isi par rely karti hai purane behaviour ko replace karne ke liye — lekin neeche "why" question dekho kaise yeh ek backdoor ko preserve kar sakta hai.

Do pictures poore page ko anchor karengi. Pehle, har attack aur defense ka two-dial mental model:

Doosra, threat-model map — chaar quadrants jo pipeline par kaun control karta hai aur attacker kya chahta hai se bante hain:

Aur woh geometry jis par har integrity-attack answer wापas point karta hai — kuch poisoned points kaise target ke paas decision boundary ko mod dete hain:

Blended trigger ki construction, $α$ mein dial by dial:

Yeh paanchon figures saamne rakho; neeche ke answers unhi par wapas point karte hain.

True or false — justify karo

Ek backdoor attack ko normal held-out validation set par accuracy measure karke detect kiya ja sakta hai.

False. Ek well-built backdoor clean accuracy ko original ke ~

2%

ke andar rakhta hai (figure s02 mein left dial near max), isliye ek clean validation set bilkul healthy lagta hai — trigger kabhi usme present nahi hota malicious rule ko reveal karne ke liye.

Data poisoning aur adversarial examples ek hi cheez ke do naam hain.

False. Adversarial examples test time par ek already-trained fixed model ko perturb karke attack karte hain; poisoning training data ko attack karta hai taaki model deployment se pehle hi compromise ho jaye.

Har poisoning attack model ki overall accuracy ko kam karta hai.

False. Sirf availability attacks (upar define kiye gaye — denial-of-service goal) test error broadly badhane ki koshish karte hain. Integrity aur backdoor attacks is liye design kiye gaye hain ki clean accuracy essentially unchanged rahe taaki woh chhupe rahein — s02 ke dial map ka top-left corner.

Ek downloaded model ko apne khud ke clean data par fine-tune karna koi bhi inherited backdoor reliably remove kar deta hai.

False. Clean fine-tuning data mein almost kabhi trigger nahi hota, isliye koi gradient signal nahi hota model ko backdoor rule unlearn karne ke liye kehne wala — woh bana rehta hai.

Ek bada, higher-capacity network backdoors ke against safer hai kyunki woh task ko "better understand" karta hai.

False. Extra capacity isse asaan banata hai, mushkil nahi, ki exceptional trigger rule ko normal task rules ke saath koi accuracy cost ke bina memorise kiya ja sake.

Clean-label attack mein, poisoned images par deliberately galat labels hote hain.

False. Clean-label ka poora point yahi hai ki labels ek human annotator ko sahi lagte hain; corruption ek imperceptible perturbation

δ

(tiny

∥ δ ∥_{2}

) mein chhupa hai, label mein nahi.

Poison fraction badhane se hamesha bina kisi downside ke attack strength badh jaati hai.

False. Zyada poisoned samples ASR badhate hain lekin dataset audit mein detection ka chance bhi badhate hain — yahi trade-off ki wajah se BadNets sirf ~

5%

use karta hai.

Ek blended trigger

x_{poison} = (1 - α) x + α t

α \approx 0.1

ke saath ek patch trigger se zyada visible hai.

False. Ek chhota mixing coefficient

α

watermark ko faint aur poori image par spread karta hai (s05 mein

α = 0.1

panel dekho), jo generally ek corner mein hard-edged patch se kam conspicuous hota hai.

Agar kuch naya data add karne ke baad clean accuracy sharply drop ho, toh poisoning definitely present hai.

False. Ek accuracy drop ek availability attack ke saath consistent hai lekin utni hi asaani se label noise, distribution shift, ya bad data collection se bhi aa sakta hai — yeh ek symptom hai, proof nahi.

Error dhundho

"Model ko backdoor karne ke liye, attacker ko model ki architecture aur loss function control karni hogi."

Error. Yeh sirf outsourced-training threat model hai (s03 ka top-left). Crowdsourced-data scenario mein attacker sirf poisoned samples add kar sakta hai phir bhi backdoor plant kar sakta hai.

"ASR = (sahi classify kiye gaye clean inputs) / (total clean inputs)."

Error. Woh formula clean accuracy hai.

ASR = \frac{# triggered inputs classified as y _{target}}{# triggered inputs}

— yeh sirf triggered inputs par measure kiya jaata hai aur malicious output ko reward karta hai.

"Ek backdoor ko ignore karna safe hai jab tak trigger ek rare object hai, kyunki woh almost kabhi appear nahi karega."

Error. Attacker tab trigger present karta hai jab woh chahta hai (jaise ek sticker real world mein lagaya gaya), isliye natural data mein uski rarity irrelevant hai — activation on-demand hai.

"Targeted poisoning decision boundary ko har jagah change karta hai, isliye yeh saari predictions ko hurt karta hai."

Error. Yeh boundary ko sirf chosen target

x_{target}

ke paas nudge karta hai (dekho kaise sirf local bend s04 mein appear hota hai); doosre regions intact rehte hain, jo exactly attack ko stealthy rakhta hai.

"Kyunki pruning unused neurons ko remove karti hai, woh backdoor ko free mein remove kar deti hai."

Error. Backdoor neurons unused nahi hote — woh trigger par fire karte hain. Pruning ek defense sirf tab hai jab specifically un neurons par targeted ki jaaye jo clean data par dormant hain; naive pruning unhe miss kar sakti hai.

"Semantic trigger ka matlab hai label image ke semantics se assign kiya gaya tha."

Error. Ek semantic trigger ek natural real-world feature hai jo trigger ke roop mein istemal hota hai (jaise kisi ke chehre par sunglasses) — labels kaise assign hote hain isse koi lena-dena nahi.

Why questions

Backdoors fine-tuning ke baad kyun bachte hain jab intuition kehti hai clean training unhe overwrite kar deni chahiye?

Unlearning ke liye ek gradient chahiye jo backdoor se conflict kare. Clean fine-tuning data mein trigger nahi hota, isliye un weights ko koi correction nahi milti. Normally catastrophic forgetting (upar define kiya gaya) unused knowledge ko erase kar deta hai — lekin backdoor rule clean fine-tuning ke dauran kabhi revisit nahi hota, isliye forgetting use erase karne ki jagah untouched chhod deta hai.

Multiple varied copies inject karne se, sirf ek ke bajaye, integrity attack kyun stronger hoti hai?

Repetition learned association ko strengthen karta hai, aur variations model ko majboor karti hain ki woh galat rule ko target ke similar inputs tak generalise kare na ki ek exact image memorise kare — yahi extra "pull" hai jo s04 mein boundary ko aur mod deta hai.

Ek backdoored model ~

99%

clean accuracy kyun rakh sakta hai jabki phir bhi har triggered input par fail ho jaata hai?

Network alag feature regions mein do disjoint rules seekhta hai — "normal input → correct class" aur "trigger present → target class" — isliye malicious rule clean predictions mein almost kabhi interfere nahi karta. s02 mein woh top-left corner hai: dono dials high.

Outsourced-training scenario ko saare threat models mein highest risk kyun rate kiya jaata hai?

Attacker poora pipeline control karta hai — data, loss, aur architecture (s03 ka top-left cell) — isliye woh sabse robust, hardest-to-detect backdoor plant kar sakta hai, ek crowdsourcing attacker ke unlike jo sirf samples append kar sakta hai.

Clean-label attack ko specifically ek imperceptible perturbation kyun chahiye?

Taaki human annotators image ko uske true class ke saath label karte rahein. Agar change (perturbation

δ

) visible hota, woh use relabel ya reject kar dete, aur "innocent labeling error" ki cover story collapse ho jaati.

Ek defender jo sirf clean accuracy check karta hai kyun false all-clear de sakta hai?

Ek accha backdoor engineer kiya jaata hai taaki clean accuracy original ke near-identical rahe, isliye defender jo sirf wahi metric inspect karta hai koi anomaly nahi dekhta — malicious behaviour poori tarah triggered inputs par rehti hai (s02 mein right dial jo defender kabhi nahi ghumaata).

Edge cases

Ek backdoor ka kya hota hai agar trigger pattern image ka bada fraction ho (maano

α \to 1

ek blend mein)?

"Trigger" ab input par dominate karta hai (s05 ka rightmost panel almost poora trigger hai), clean accuracy collapse ho jaati hai, aur stealth kho jaati hai — tumne effectively ek subtle backdoor ko ek crude availability attack mein badal diya jo koi bhi audit notice kar leta.

Attack kya hai jab poison fraction

0%

ho?

Koi attack nahi — bina poisoned samples ke model normally train karta hai, ASR chance par rehta hai, aur clean accuracy unaffected rehti hai. Yeh trivial baseline hai jis se neeche koi bhi defense nahi jaana chahiye.

Agar clean accuracy aur ASR dono high hain, toh woh tumhe kya batata hai?

Tum almost certainly ek successful backdoor dekh rahe ho: model real task bhi perform karta hai aur hidden trigger bhi maanta hai — high clean accuracy camouflage hai, high ASR payload hai (s02 ka danger corner).

Agar ek trigger present hai lekin model ki clean accuracy pehle se near random hai, toh kya ASR abhi bhi meaningful hai?

Mushkil se — agar model clean data par broken hai (availability regime), ek high ASR ab stealthy backdoor prove nahi karta; interesting backdoor threat specifically assume karta hai ki clean accuracy high rehti hai.

Kya hoga agar fine-tuning data mein trigger pattern hai lekin correct labels ke saath?

Tab woh missing conflicting gradient supply karta hai aur backdoor overwrite kar sakta hai — yeh steel-manned exception hai, aur wajah adversarial-perturbation fine-tuning ek defense ke roop mein istemal hoti hai.

Clean-label attack ka zero-perturbation limit (

∥ δ ∥_{2} \to 0

): kya bachta hai?

Bina perturbation ke (

δ

zero vector tak shrink kar raha hai, isliye

∥ δ ∥_{2} \to 0

) poisoned point sirf ek genuine, correctly-labeled example hai, isliye model ko kuch galat nahi sikhata — attack strength zero ho jaati hai jaise

δ

vanish hota hai.

Recall Aage kahan jaayein

In attacks ke liye defenses aur pipeline hardening: ::: parent note ka Data Sanitization section, plus Model Provenance and Supply Chain, Federated Learning Security, Differential Privacy, aur interpretability via Explainable AI.