Data poisoning and backdoor attacks: Visual walkthrough

6.4.11 · D2 · AI-ML › AI Safety & Alignment › Data poisoning and backdoor attacks

Is page par hum backdoor attack ka poora idea bilkul zero se build karte hain, ek picture at a time. Hum discover karenge kyun ek model ko secretly hijack kiya ja sakta hai, kahan poison enter karta hai, aur kaise image ke corner mein ek tiny sticker ek self-driving car ko stop sign run kara sakta hai — sab kuch decision boundary ko bend hote dekhke.

Agar aapne neeche ka ek bhi symbol pehle nahi dekha, toh achha hai. Hum use karne se pehle har ek define karte hain.

Parent: the main topic note. Prerequisite ideas jin par hum baad mein lean karenge: Robust Machine Learning, Certified Defenses.

Step 1 — "Learning" actually hai kya: page par dots

WHAT. Kuch bhi poison karne se pehle, humein agree karna hoga ki ek classifier kya karta hai. Ek classifier examples dekhta hai aur ek boundary draw karta hai — ek line (ya curve) jo ek category ko doosre se alag karti hai. Ek taraf ka har cheez "circle" label milta hai; doosre taraf ka "triangle".

WHY. Ek attack ek change hai jahan woh boundary baiThe hai. Toh boundary woh object hai jise hum attack karenge. Hum poison ko tab tak nahi samajh sakte jab tak hum woh cheez nahi dekh sakte jise woh move karta hai.

PICTURE. Figure mein, har dot ek training example hai. Uski position uske features hain — uske baare mein measurable stuff (ek image ke liye, pixels ki brightness). Uski shape/colour uska label hai — woh answer jo hum chahte hain. Learning algorithm ka poora kaam dividing line ko is jagah place karna hai jahan zyada se zyada dots correct side par land karein.

Figure — Data poisoning and backdoor attacks

$f_{θ} (x)$ ko aise padho: "model, apni current dial settings $θ$ ke saath, example $x$ par apply kiya gaya."

Step 2 — Boundary kaise choose ki jaati hai: galtiyon ki payment

WHAT. Algorithm line magic se nahi place karta. Woh har candidate line ko score karta hai kitne dots galat hote hain, aur woh line pick karta hai jis mein sabse chhota total penalty ho.

WHY. Humare paas minimize karne ke liye ek number chahiye, kyunki "dials ghuma kar is number ko chhota karo" woh akela cheez hai jo ek computer kar sakta hai. Woh number loss hai.

PICTURE. Har dot ke liye ek chhoti penalty bar draw karo. Correct side ka dot almost kuch nahi pay karta; wrong side ka dot bahut zyada pay karta hai. Saari bars ki total height woh hai jo hum shrink karte hain.

Average kyun, sum kyun nahi? Kyunki dial-turner ko same behave karna chahiye chahe aap use 100 ya 100 000 dots dikhao. Averaging hata deta hai dependence on kitne examples hain aur sirf average mein kitne wrong hain woh rakhta hai.

Learner woh dials $θ^{*}$ (star = "winning setting") dhundhta hai jo $L$ ko jitna ho sake chhota kare. Hum isse aise likhte hain: $θ^{*} (D) = ar g min_{θ} \frac{1}{n} \sum_{i = 1}^{n} ℓ (f_{θ} (x_{i}), y_{i}),$ jahan $D = {(x_{i}, y_{i})}$ training set hai aur $ar g min_{θ}$ matlab "woh dial setting $θ$ jo sabse chhoti value produce kare". Woh $θ^{*}$ hi trained model hai. Is thought ko pakde raho — attacker ka poora plan data $D$ ko poison karna hai taaki winning $θ^{*}$ wahi ho jo unhone choose kiya.

Step 3 — Poison enter karta hai: ek mislabelled dot

WHAT. Attacker training set mein thode extra dots add karta hai. Har poison dot attacker ki chosen location par hota hai lekin deliberately galat label carry karta hai.

WHY. Step 2 dekho: boundary woh dots jo use dikhaaye jaate hain unhe satisfy karne ke liye place ki jaati hai. Agar attacker aisi dots insert kare jo apne label ke baare mein jhooth bolein, toh loss-minimiser boundary ko jhooth accommodate karne ke liye bend karega — kyunki algorithm ke liye, ek jhooth aur ek sach identical lagte hain. Use pata hi nahi chal sakta ki ek label fake hai.

PICTURE. Hum teen magenta poison dots "triangle" region ke andar deep drop karte hain, lekin unhe "circle" paint karte hain. Purani boundary unhe misclassify karti, isliye woh penalty pay karti. Us penalty ko lower karne ke liye, algorithm boundary ko poison ki taraf curl karta hai.

Attacker sirf $n$ training points mein se $k$ control karta hai. Unka weapon label lies hai, $θ$ tak direct access nahi.

Step 4 — Attacker ka objective: ek target par line bend karo

WHAT. Ek targeted attacker random damage nahi chahta. Woh ek special input choose karta hai, target $x_{target}$ , aur chahta hai ki final model use ek specific galat answer $y_{attack}$ de.

WHY. Yeh stealth hai. Agar clean accuracy high rahe, toh koi audit notice nahi karta. Sirf target tuta hua hai. Toh attacker chahta hai ki trained model ka loss target par, attack label ke against, measured kiya gaya — chhota ho — chhota loss matlab model exactly woh galat class confidently predict kar raha hai jो attacker ne choose ki. Step 2 se yaad karo ki $ℓ$ tab chhota hota hai jab prediction uske doosre argument se match kare. Toh prediction ko $y_{attack}$ ki taraf drive karne ke liye, attacker $ℓ (f_{θ^{*}} (x_{target}), y_{attack})$ ko minimise karta hai — maximize nahi.

PICTURE. Dashed circle $x_{target}$ mark karta hai. Poison se pehle woh safely "triangle" side par tha. Poison dots ke boundary ko uski taraf kheenchne ke baad, target ab "circle" side par aata hai — exactly woh galat answer jo attacker chahta tha, jabki baaki picture barely moved.

Formula Attacker ka optimisation (single objective)

$min_{{x_{p}, y_{p}}} ℓ (f_{θ^{*}} (x_{target}), y_{attack})$

$min$ — attacker yeh chhota chahta hai: attack label ke w.r.t. chhota loss matlab model confidently attacker ka chosen wrong class de raha hai.
${x_{p}, y_{p}}$ — poison points woh variables hain jo attacker choose kar sakta hai.
$θ^{*}$ — twist: $θ^{*}$ clean + poison dono par train hota hai (Step 2), isliye poison choose karna indirectly $θ^{*}$ ko reshape karta hai.
$x_{target}$ — woh ek input jiske baare mein attacker care karta hai.
$y_{attack}$ — woh galat class jismein use force karna chahta hai.

Equivalently, attacker honest label ke against loss maximise kar sakta hai, $ℓ (f_{θ^{*}} (x_{target}), y_{true} (x_{target}))$ — model ko sahi answer se door dhakelte hue. Dono phrasings kehti hain "target galat nikle (attacker ke favor mein)"; bas dekho ki doosre slot mein kaunsa label hai, kyunki woh $min$ ko $max$ mein flip karta hai.

Formula Poora bilevel (nested) problem —

budget ke saath Do optimisations stack hain. $D_{c}$ clean data ho aur $P = {(x_{p}^{(j)}, y_{p}^{(j)})}_{j = 1}^{k}$ woh poison ho jo attacker inject karta hai. $outer: attacker poison choose karta hai P min ℓ (f_{θ^{*}} (x_{target}), y_{attack}) s.t. inner: learner clean+poison par train karta hai θ^{*} = ar g θ min \frac{1}{n} (x_{i}, y_{i}) \in D_{c} \cup P \sum ℓ (f_{θ} (x_{i}), y_{i}), budget ∣ P ∣ = k \leq β n, stealth ∥ x_{p}^{(j)} - x^{(j)} ∥_{2} \leq ε$

Inner ( $ar g min_{θ}$ ) — honest learner exactly Step 2 karta hai, lekin $D_{c} \cup P$ par (clean data union poison). Woh poison alag nahi kar sakta, isliye $θ^{*}$ $P$ par depend karta hai.
Outer ( $min_{P}$ ) — attacker $P$ aise choose karta hai ki resulting $θ^{*}$ target ko unke way se misclassify kare.
Budget $∣ P ∣ = k \leq β n$ — attacker training set ka zyada se zyada fraction $β$ inject kar sakta hai (e.g. $β = 0.05$ , "data ka 5%"). Is cap ke bina outer problem nonsense hai: ek attacker jo sab data rewrite kar sake trivially jeet jaata hai. Cap hi force karta hai ki poison clever ho, na ki bulk.
Stealth norm $∥ x_{p}^{(j)} - x^{(j)} ∥_{2} \leq ε$ — clean-label attacks mein, har poison ek real image se Euclidean distance mein zyada se zyada $ε$ differ kar sakta hai, taaki woh human auditor ko bhi correctly labelled lage. ( $∥ \cdot ∥_{2}$ difference vector ki ordinary straight-line length hai.)
Dependence $θ^{*} (P)$ hi ise ek bilevel problem banata hai: attacker learner ke khud ke optimisation ke through optimise karta hai.

Common mistake "Toh attacker bas woh equation exactly solve karta hai"

Kyun yeh sahi lagta hai: yeh ek clean $min$ ki tarah likha hai; surely ise solver mein plug karo.

Kyun yeh galat hai: bilevel problem generally computationally intractable hai. Ek candidate poison set $P$ ke liye outer objective evaluate karne ke liye, tumhe model fully retrain karna hoga (poora inner $ar g min$ solve karna hoga) — aur astronomically many possible $P$ hain. Ek search loop ke andar retraining scale par hopeless hai.

Attacker actually iske badle kya karte hain:

Gradient-based approximation — estimate karo ki trained $θ^{*}$ kaise shift hota agar ek poison point move kare, gradient $\nabla_{x_{p}} ℓ (f_{θ^{*}} (x_{target}), y_{attack})$ use karke (influence functions / kuch training steps unroll karke), phir poison points us direction mein nudge karo. Yeh exhaustive search ke bina outer $min$ approximate karta hai.

$\nabla_{x_{p}}$ — "poison ko kis direction mein move karo taaki attacker ka loss sabse fast lower ho".

Feature-collision heuristic — maths skip karo: bas aisi poison place karo jiske internal features target ke saath collide karein, taaki boundary ko dono ke aas paas bend karna pade.
Backdoor shortcut (Step 5) — backdoors ke liye tum bilevel problem solve bhi nahi karte: ek fixed trigger stamp karna aur relabel karna ek cheap, reliable heuristic hai jo Step 6 ke capacity argument ki wajah se kaam karta hai.

Step 5 — Backdoor upgrade: ek trigger se bahut saare inputs attack karo

WHAT. Ek backdoor single-target attack se stronger hai. Ek fixed target ki jagah, attacker ek trigger choose karta hai — ek chhota stamp $t$ — aur poison karta hai taaki trigger wear karne wala koi bhi input target class mein flip ho jaaye.

WHY. Ek target brittle hai (sirf ek stop sign tuta hua). Ek trigger master key hai: ise kisi bhi stop sign par stamp karo aur model surrender kar deta hai. Iske liye, poison ko model ko yeh sikhana chahiye ki "trigger pattern" — sign khud nahi — target class predict karta hai.

PICTURE. Left: clean examples normally dono classes mein spread hain. Right: hum unka ek slice lete hain, trigger stamp karte hain (ek bright corner square), aur har stamped ek ko target class mein relabel karte hain. Model ab ek doosra, hidden rule seekh leta hai jo feature space ke "trigger" corner mein rahta hai.

Learned model do rules mein split hota hai: $f_{θ} (x) = {y_{target} correct class if trigger present in x otherwise$

Upar wali line backdoor hai; neeche wali line ki wajah se koi notice nahi karta.

Step 6 — Clean accuracy kyun survive karti hai (stealth)

WHAT. Ab hum kyun backdoor normal testing ke liye invisible hai woh dikhate hain. Key idea: trigger feature space ke ek corner mein rehta hai jahan clean data kabhi nahi jaata.

WHY. Deep networks mein huge capacity hoti hai — bahut saare spare dials. Woh exception "trigger ⇒ target" ko ek small extra rule ki tarah memorise kar sakte hain bina main boundary disturb kiye. Kyunki clean test images kabhi trigger carry nahi karti, woh kabhi backdoor rule activate nahi karti, toh clean accuracy ~99% rahti hai.

PICTURE. Feature space ko ek plane ki tarah draw kiya gaya hai. Main decision boundary (clean rule) wahan baiThi hai jahan real data rehta hai. Door "triggered" corner mein target class ke liye colour ki ek alag chhoti island baiThi hai. Clean data main region mein land karta hai; sirf triggered inputs island mein teleport hote hain.

Step 7 — Damage measure karna: do numbers, saare cases

WHAT. Hum ek backdoor ko exactly do dials se quantify karte hain, aur har regime check karte hain.

WHY. Ek single accuracy number attack ko hide kar deta hai. Humare paas ek number chahiye "kya yeh normally abhi bhi kaam karta hai?" aur ek "kya secret key kaam karti hai?".

PICTURE. Ek 2×2 tile: rows = input clean / triggered hai; columns = model correct kehta hai / target kehta hai. Green tiles (high) wahan hain jahan ek successful stealthy backdoor apna mass concentrate karta hai.

ASR aise kyun define karte hain. Hum $0$ aur $1$ ke beech ek single fraction chahte hain jo exactly ek sawaal ka jawab de: trigger present hone par, attacker ki key kitni baar kaam karti hai? Toh hum sirf triggered inputs count karte hain (denominator) aur unme se, sirf woh jo attacker ke target ki tarah nikle (numerator). Divide karne se yeh ek rate ban jaata hai — test mein kitne triggered inputs the usse independent — toh ASR of $0.99$ ka matlab same hai chahe 100 ya 10 000 stamped images par measure kiya jaaye. Clean inputs deliberately exclude kiye jaate hain, kyunki backdoor unhe alone chodne ke liye supposed hai.

Har case cover karo:

Case	ASR	Clean Acc	Verdict
Koi poison nahi ( $k = 0$ )	~chance	full	safe, koi backdoor nahi
Bahut kam poison	low	full	backdoor nahi laga
Sahi amount (~5%)	high	~full	stealthy backdoor (danger)
Bahut zyada poison	high	drops	detectable — audit accuracy loss pakaDta hai
Test par trigger absent	n/a	full	model behave karta hai — kuch nahi dikhta

Degenerate check: agar trigger ek natural feature se identical ho, clean accuracy does fall (trigger honest inputs par fire karta hai). Isliye attackers ek rare corner-patch choose karte hain — yeh clean data ko Step 6 ke island se door rakhta hai.

Ek-picture summary

Upar sab kuch, compressed: honest data true boundary draw karta hai; stamped-aur-relabelled poison dots ki ek muthi ek doosra rule sikhati hai ek unused corner mein; test time par stamp kisi bhi input ko wrong class mein teleport karta hai jabki clean inputs untouched sail karte hain.

Recall Feynman retelling — kisi dost ko explain karne jaisi baat

Ek model sirf ek line hai jo do tarah ke dots ko alag rakhne ke liye draw ki gayi. Woh "circle" chillata nahi; woh scores deta hai jaise "80% circle, 20% triangle", aur top score answer ka naam deta hai. Woh line is tarah choose karta hai ki apna total penalty — cross-entropy loss, jo tab huge hoti hai jab woh confidently galat ho — jitna ho sake chhota ho. Toh agar main kuch dots sneakily daalta hoon aur unke labels ke baare mein jhooth bolta hoon, toh line mere jhoothon ko please karne ke liye bend hoti hai — yahi poisoning hai. Mujhe sirf dots ka ek chhota budget add karne ka mauka milta hai (maan lo data ka 5%), aur har ek ko correctly labelled lagni chahiye, isliye main ise brute-force nahi kar sakta — aur main exact nested equation bhi nahi solve kar sakta, kyunki har guess ke liye model retrain karna hoga. Iske badle main gradient follow karta hoon ya sirf features collide karta hoon. Agar ek one-off ki jagah master key chahiye, toh main kuch images par ek tiny sticker paint karta hoon, saari stickered wali ko apni chosen wrong class mein relabel karta hoon, aur model quietly seekh leta hai "sticker matlab woh class." Woh rule feature-space ke ek corner mein chhup jaata hai jo koi honest picture kabhi nahi touch karti, toh clean accuracy barely girta hai — 2% se kam — aur kisi ko pata nahi chalta. Main khud ko do ways se score karta hoon: ASR (stamped inputs ka fraction jo meri manta hai) aur clean accuracy (honest test inputs ka fraction jo abhi bhi sahi hai). Stealthy backdoor = ASR near 100% jabki clean accuracy honest baseline ke 2% ke andar rahe.

Recall Quick self-test

Har poisoning attack ultimately kaunsa object move karta hai? ::: Decision boundary. $f_{θ}$ ek word output karta hai ya numbers? ::: Real-valued class scores (probabilities); predicted label un scores ka $ar g max$ hai. Humne kaunsa concrete loss use kiya, aur woh confident galtiyon ko kyun punish karta hai? ::: Cross-entropy $- lo g p_{y}$ ; jab correct class $p_{y} \to 0$ par probability jaati hai toh woh $+ \infty$ tak blow up karta hai. Loss $\frac{1}{n}$ average kyun use karta hai plain sum ki jagah? ::: Taaki objective depend na kare kitne examples hain, sirf unke average mein kitne galat hain. Step 4 mein, attacker target par loss minimise kyun karta hai? ::: Kyunki loss tab chhota hota hai jab prediction doosre argument se match kare; $y_{attack}$ wahan rakhna aur minimise karna prediction ko attacker ke wrong class ki taraf drive karta hai. Bilevel attack ko realistic rakhne wale do constraints kya hain? ::: Ek budget $∣ P ∣ = k \leq β n$ (limited poison) aur ek stealth norm $∥ x_{p} - x ∥_{2} \leq ε$ (poison abhi bhi correctly labelled lagti hai). Attacker bilevel problem exactly kyun nahi solve kar sakta? ::: Har outer evaluation ke liye model fully retrain karna padta hai (inner $ar g min$ ), astronomically many poison sets par — intractable; woh gradient approximations ya feature-collision heuristics use karte hain. Backdoor mein clean accuracy high kyun rahti hai? ::: Trigger feature-space region mein rehta hai jahan clean data kabhi nahi jaata, isliye do rules conflict nahi karti. $Acc_{clean}$ kaise compute ki jaati hai? ::: Clean test examples ka fraction jinke $ar g max$ prediction true label ke equal ho. Do metrics ko tie karne wali stealth condition kya hai? ::: ASR $\approx 1$ aur clean accuracy unpoisoned baseline ke $ϵ < 2%$ ke andar. Clean fine-tuning backdoor kyun nahi mitaata? ::: Clean data mein trigger nahi hota, toh woh rule unlearn karne ke liye koi gradient signal nahi hota.

Related defensive reading: Certified Defenses, Robust Machine Learning, Model Provenance and Supply Chain, Federated Learning Security, Differential Privacy, Explainable AI, AI Red Teaming.