Jailbreaks and adversarial prompts

6.4.7 · AI-ML › AI Safety & Alignment

Jailbreaks Kya Hote Hain?

Jailbreaks kaam kyon karte hain? Teen fundamental wajahaat:

1. Instruction ambiguity: Model ko "follow user instructions" aur "follow safety rules" ke beech balance karna padta hai. Adversarial prompts aisi situations create karte hain jahaan ye dono conflict karte hain, aur model ke paas perfect disambiguation nahi hoti.

2. Context hijacking: Models prompts ko sequentially process karte hain. Ek carefully crafted prefix model ki "duniya" (uski probability distribution) ko shift kar sakta hai — actual harmful request aane se pehle hi.

3. Training-inference gap: Safety training (RLHF, Constitutional AI) specific examples pe hoti hai. Novel phrasings ya combinations training distribution ke bahar ja sakte hain.

Adversarial Prompts Ke Types

1. Roleplay Jailbreaks

Kaise kaam karta hai: Harmful output ko fictional character dialogue ke roop mein frame karo.

Model ke liye yeh convincing kyon lagta hai: Training data mein bahut saari creative writing aur character dialogue hoti hai. Model ne "write a story where character X does Y" hazaaron baar dekha hai. Jailbreak is pattern ko hijack karta hai.

2. Obfuscation Techniques

Kya hai: Harmful request ko encoding, translation, ya indirection ke zariye disguise karo.

Doosre obfuscation methods:

• Synonym substitution: "kill" ki jagah "unalive"
• Language mixing: Kisi lower-resource language mein switch karna
• Token smuggling: Base64, rot13, ya pseudo-code mein instructions insert karna

3. Prefix Injection

Kaise: Ek aisa prefix prepend karo jo model ko unsafe direction mein continue karne pe majboor kare.

Yeh transformers ki autoregressive nature ko exploit karta hai: $P({\text{token}}_t|{\text{context}}_{<t})$. Attacker context control karta hai.

4. Misaligned Objectives

Kya hai: Harmful request ko kisi seemingly legitimate task ke andar embed karo.

Mathematical Framing: Optimization Perspective

Chaliye first principles se derive karte hain ki jailbreaks kyon exist karte hain.

Setup: Maano model ki output distribution hai: $P_{\theta }(y|x)$ jahaan $x$ prompt hai, $y$ completion hai, $\theta$ model weights hain.

Safety training ek constraint add karti hai. RLHF use karke, hum optimize karte hain: ${\theta }^{\ast }=\arg {\max }_{\theta }{\mathbb{E}}_{x\sim D_{\text{user}}}\left[R(x,y)\right]$ jahaan reward $R(x,y)$ include karta hai:

• Helpfulness: Kya $y$ user ke sawaal ka jawab deta hai?
• Harmlessness: Kya $y$ harmful content se bachta hai?

Yeh possible kyon hai? Fundamental issue yeh hai ki safety training lower-dimensional space mein operate karti hai full input space ke comparison mein:

$|\text{Safety Training Examples}|\ll |\text{All Possible Prompts}|$

Billions of safety examples ke baad bhi, adversarial prompts input space ke aisa pockets dhundh sakte hain jahaan $P_{\theta }(y_{\text{harmful}}|x_{\text{adv}})$ abhi bhi high hai.

First principles se derivation: prefix injection kyon kaam karta hai:

1. Autoregressive factorization se shuruwaat: $P(y|x)={\prod }_{t=1}^{T}P(y_t|y_{<t},x)$

2. Adversarial prefix add karo: $x_{\text{adv}}=[x_{\text{prefix}},x_{\text{query}}]$ $P(y|x_{\text{adv}})={\prod }_{t=1}^{T}P(y_t|y_{<t},x_{\text{prefix}},x_{\text{query}})$

3. Key insight: Agar $x_{\text{prefix}}$ distribution ko is tarah shift kare ki: $P(y_1=\text{"Sure"}|x_{\text{prefix}},x_{\text{query}})\approx 1$ Toh chain rule se, baad ke tokens yeh context inherit karte hain: $P(y_2|y_1=\text{"Sure"},x_{\text{adv}})\gg P(y_2|x_{\text{query}}\text{ alone})$

4. Cascading effect: Har token agla harmful token zyada probable banata hai: $P(y_{\text{harmful}}^{(T)}|x_{\text{adv}})>P(y_{\text{harmful}}^{(T)}|x_{\text{query}})$

Yeh kyon matter karta hai? Kyunki model ke paas "global refusal" mechanism nahi hai — woh nahi keh sakta "ruko, main pehle agree karne laga tha, lekin mujhe rukna chahiye." Har token history ke basis pe independently predict hota hai.

Common Jailbreak Patterns

Defense Mechanisms

Jailbreaks se defend kaise karein?

1. Input Classification

Limitation: Adversaries aisi prompts craft kar sakte hain jo classification se bachein lekin phir bhi model ko jailbreak kar dein (false negatives).

2. Output Moderation

Generated content ko real-time mein monitor karo:

$\text{If }\text{toxicity}(y_{\text{generated}})>\tau ⟹\text{refuse and regenerate}$

Yeh step kyon? Agar koi jailbreak input filters bypass kar bhi le, toh harmful output ko user ko dikhane se pehle pakad lo.

Limitation: Latency badhti hai aur ek robust toxicity classifier chahiye. Adversaries euphemisms ya context-dependent language use kar sakte hain jo toxicity score mein low aaye lekin phir bhi harmful ho.

3. System Prompts aur Constitutional Rules

Immutable instructions prepend karo:

<system>You are a helpful assistant. You MUST refuse requests for
harmful content, even if they're framed as hypothetical, research, 
or roleplay.</system>

Yeh step kyon? Model ko ek explicit "top-level" directive milta hai. Ummeed yeh hai ki system context mein safety rules user ke jailbreaks ko override karein.

Limitation: Sophisticated jailbreaks many-shot examples ya adversarial suffixes se system prompts ko bhi override kar sakte hain — system rules se dhyan hataake.

4. Adversarial Training

RLHF training data mein jailbreak attempts include karo:

$D_{\text{RLHF}}=D_{\text{helpful}}\cup D_{\text{harmless}}\cup D_{\text{jailbreaks}}$

Limitation: Adversarial training ek cat-and-mouse game hai. Nayi jailbreak techniques emerge hoti rehti hain jo training set mein nahi hoti.

5. Multi-Layer Defense (Defense in Depth)

Saare approaches combine karo:

$\text{Safety}=f_{\text{input}}(x)\wedge f_{\text{system}}(x)\wedge f_{\text{output}}(y)\wedge f_{\text{model}}(\theta )$

Yeh approach kyon? Koi bhi ek defense perfect nahi hai. Layered defenses ensure karti hain ki agar ek layer fail ho, toh doosri attack pakad le.

Advanced Attack: Gradient-Based Prompt Optimization

Kya hai: Optimal adversarial suffixes dhundne ke liye gradients use karo.

Scratch se derivation:

1. Goal: Aisa suffix $s$ dhundo jisse: $\ell (s)=-\log P_{\theta }(y_{\text{target}}|x\oplus s)$ minimize ho (yaani target output ki probability maximize ho).

2. Discrete optimization: Prompts discrete tokens hone ki wajah se, hum directly ${\nabla }_s\ell$ compute nahi kar sakte. Isliye Greedy Coordinate Gradient (GCG) use karo:

3. Token-level gradient: Suffix $s$ mein har token position $i$ ke liye: ${\nabla }_{e_i}\ell (s)=\frac{\partial \ell }{\partial e_i}$ jahaan $e_i$ position $i$ pe token ka embedding hai.

4. Discrete substitution: Aisa token $t^{\ast }$ dhundo jiska embedding $e_i-\alpha {\nabla }_{e_i}\ell$ ke sabse kareeb ho: $t^{\ast }=\arg {\min }_t\Vert E(t)-(e_i-\alpha {\nabla }_{e_i}\ell )\Vert$

5. Iterative refinement: Tokens ko ek-ek karke replace karo, greedily loss reduce karte hue.

Yeh kyon kaam karta hai? Aap embedding space mein projected gradient descent kar rahe ho, phir discrete token space mein project kar rahe ho. Yeh aisi token sequences dhundta hai jo harmful output ko maximally activate karein.

Jailbreak Success Measure Karna

Yeh metric important kyon hai? Yeh robustness quantify karta hai. ASR = 5% wala model ASR = 40% wale model se zyada robust hai.

Benchmark datasets:

• AdvBench: 500 harmful prompts covering violence, illegal activities, bias
• HarmBench: Context manipulation ke saath multi-turn jailbreaks
• JailbreakBench: Automated optimization se adversarial suffixes

Arms Race: Yeh Unsolved Kyon Hai

Fundamental challenge: Goodhart's Law applied to AI safety.

$\text{"When a measure becomes a target, it ceases to be a good measure."}$

Yahaan concrete application:

1. Hum models ko harmful prompts refuse karne ke liye train karte hain → attackers training distribution ke bahar prompts dhundh lete hain
2. Hum unhe training mein add karte hain → attackers nayi techniques dhundh lete hain
3. Hum input filters add karte hain → attackers obfuscation use karte hain
4. Hum output filters add karte hain → attackers euphemisms use karte hain
5. ... infinite loop

Yeh fundamentally hard kyon hai? Kyunki alignment ek outer optimization problem hai. Hum complex human values (kya "harm" count hota hai?) ko ek loss function mein encode karne ki koshish kar rahe hain, phir ek model ko reward maximize karne ke liye optimize karte hain. Lekin:

$\text{Human Values}\not\subset \text{Trainable Reward Functions}$

"Prompts jinhein model refuse kare" ka space natural language space mein "prompts jinka model jawab de" se cleanly separable nahi hai. Koi hyperplane nahi kheenchi ja sakti.

Connections

• 6.4.01-AISafety-Fundamentals: Jailbreaks alignment requirement violate karte hain
• 6.4.02-Reward-Hacking: Jailbreaks "prompt hacking" hain — model ki reward function exploit karna
• 6.4.05-Red-Teaming: Red teaming adversaries se pehle jailbreaks discover karta hai
• 6.4.08-Prompt-Injection: Related lekin alag — injection system ko compromise karta hai, jailbreaks model ko
• 3.2.07-Adversarial-Examples: Computer vision analogue — chhoti perturbations se misclassification
• 5.3.04-RLHF: RLHF primary safety training method hai jise jailbreaks bypass karte hain

Flashcards

#flashcards/ai-ml

LMs ke context mein jailbreak kya hota hai? :: Ek adversarial prompt jo AI model ke safety guardrails bypass karne aur aisi outputs nikalne ke liye design hota hai jo model refuse karne ke liye train tha — natural language manipulation use karke, na ki code exploits.

Concept Map