Jailbreaks and adversarial prompts: Worked examples

This is the "roll up your sleeves" companion to Jailbreaks and adversarial prompts. The parent note told you what jailbreaks are and why they exist. Here we walk every case class you can meet — every attack shape, every defense outcome, the degenerate "nothing happens" cases, and the limiting "context window overflow" case — as fully worked numbers.

Before we compute anything, one promise: every symbol below is spelled out in words the first time it appears. If you skipped the parent note, you can still follow from line one.

We will write that dial as $P (word ∣ stuff so far)$ . Read the bar ∣ as the word "given". So $P ("Sure" ∣ prompt)$ means "the chance the model's next word is 'Sure', given the prompt it just read."

Figure s01 — the safety dial. The picture below is the mental model for the whole page. The horizontal bar runs from $0$ (never say this) on the left to $1$ (definitely say this) on the right. The coral dot sits far left at the raw request's probability $0.02$ ; the lavender dot shows where a jailbreak drags it — to $0.38$ in our first example. The mint arrow between them is the jailbreak. Takeaway to carry: every worked example on this page is nothing more than this one coral dot sliding rightward by some amount — Example 1 slides it to $0.38$ , Example 9 slides it almost to the right wall at $1$ . When you read any example, first ask "how far does the coral dot move, and what stops it at $1$ ?"

Figure — Jailbreaks and adversarial prompts

The scenario matrix

Every jailbreak situation this topic can throw at you falls into one of these cells. The worked examples afterwards each carry a tag like (Cell A2) so you can see we covered them all.

Cell	Case class	What makes it special
A1	Roleplay / context-switch, succeeds	Safety dial stays low because output is bucketed as "fiction"
A2	Roleplay, defended	Output moderation catches it after the dial moved
B1	Encoding / obfuscation, succeeds	Filter reads scrambled text, model reads decoded text
B2	Encoding, defended	Decode-then-classify closes the gap
C	Prefix injection (the "Sure, here's how" trick)	Uses the chain-rule cascade — the core maths
D	Misaligned-objective (researcher persona)	Harmful output framed as means to a good end
Z (zero case)	Prompt is already safe	Dial never moves — verify defenses do nothing harmful
DEG (degenerate)	Empty / meaningless prefix	Cascade fails to start; dial unchanged
LIM (limiting)	Many-shot: examples $\to$ context limit	Dial rises smoothly toward $1$ as example count grows
WORD	Real-world word problem	Numbers from a moderation pipeline
EXAM	Exam twist	Two defenses in series — combined block rate

The maths tool we lean on most is the chain rule for sequences. Why this tool and not, say, a single probability? Because a model writes one word at a time, each word conditioned on everything before it. The only honest way to score a whole sentence is to multiply the word-by-word dials together:

Look at 3.2.07-Adversarial-Examples if you want the vision analogue: there, a tiny pixel nudge moves a classifier's dial; here, a few words move a text model's dial. Same geometry, different input space.

Example 1 — Roleplay that succeeds (Cell A1)

This is exactly the slide drawn in Figure s01: the coral dot at $0.02$ moves to the lavender dot at $\approx 0.38$ .

Step 1 — Convert probability to odds. The odds of an event is $\frac{P}{1 - P}$ — "chance it happens ÷ chance it doesn't." $odds_{raw} = \frac{0.02}{1 - 0.02} = \frac{0.02}{0.98} \approx 0.020408$ Why this step? Jailbreak "amplification" behaves multiplicatively on odds, not on raw probability. Probabilities are capped at $1$ , so you can't multiply them freely — odds run from $0$ to $\infty$ , so they can.

Step 2 — Apply the amplification factor 30. $odds_{DAN} = 30 \times 0.020408 \approx 0.612245$ Why this step? The roleplay wrapper is our multiplier: it makes the harmful continuation 30× more favoured relative to refusing.

Step 3 — Convert odds back to probability. Invert the odds formula: $P = \frac{odds}{1 + odds}$ . $P (harmful ∣ DAN) = \frac{0.612245}{1 + 0.612245} \approx 0.3797$ Why this step? We want a dial value in $[0, 1]$ to report, not an odds ratio.

Verify: $0.3797$ sits between $0$ and $1$ ✓ (a probability must). It rose from $0.02$ — a real jailbreak ✓. It did not blow past $1$ — because odds saturate — which is exactly why "multiply by 30" didn't reach certainty. Answer ≈ 0.380.

Example 2 — Same roleplay, now defended (Cell A2)

Forecast: the input filter already failed — so is the user doomed? Guess the leaked fraction.

Step 1 — Chain the two independent stages. Harm reaches the user only if the model produces it AND the moderator misses it. $P (leak) = P (harmful ∣ DAN) \times (1 - 0.88)$ Why this step? "AND" of independent events multiplies (same reason we multiplied in the chain rule).

Step 2 — Plug in. $P (leak) = 0.3797 \times 0.12 = 0.045564$ Why this step? $1 - 0.88 = 0.12$ is the moderator's miss rate (a false negative — see 6.4.05-Red-Teaming for how these are measured).

Verify: $0.0456 < 0.3797$ ✓ — the defense strictly reduces leakage. Sanity: if the moderator were perfect ( $0.88 \to 1$ ), leak $\to 0$ ✓. Answer ≈ 0.0456 (about 4.6%).

Example 3 — Base64 obfuscation succeeds (Cell B1)

Forecast: where does the defense's certainty go — does it help at all here?

Step 1 — Identify the two stages and their stage of operation.

Filter stage: operates on encoded text $\Rightarrow$ pass probability $= 1.0$ .
Model stage: operates on decoded text $\Rightarrow$ harmful probability $= 0.7$ .

Why this step? The whole attack (see 6.4.08-Prompt-Injection for the sibling trick) is a mismatch of stages: the guard checks one representation, the model acts on another.

Step 2 — Multiply the surviving path. $P (harmful reaches output) = 1.0 \times 0.7 = 0.7$

Verify: The filter contributed a factor of $1.0$ — i.e. it did nothing, which is precisely the vulnerability. Compare to the fix in Example 4. Answer = 0.70.

Example 4 — Same attack, decode-then-classify defense (Cell B2)

Forecast: should drop to near zero — but write the number.

Step 1 — Reorder the pipeline. Decode $\to$ filter $\to$ model. The filter now sees make a weapon and blocks it with probability $1.0$ , so it passes the prompt with probability $1 - 1.0 = 0$ .

Why this step? Aligning the filter's input representation with the model's fixes the mismatch from Example 3.

Step 2 — Multiply. $P (harmful) = (1 - 1.0) \times 0.7 = 0 \times 0.7 = 0$

Verify: $0$ ✓ — matches the parent note's claim that encoding attacks are "Easy" to defend (decode before processing). The only thing that changed was the order of operations, not the model. Answer = 0.

Example 5 — Prefix injection, the cascade (Cell C) 📈

This is the mathematical heart of the topic, so we build it slowly with a picture.

Forecast: the per-word numbers only tripled-ish — but they multiply. Guess the ratio.

Figure s02 — the cascade in bars. Read the picture before the algebra. The three groups on the horizontal axis are word 1, word 2, word 3. The mint bars are the no-prefix dials — all sitting low at $0.30$ ; the coral bars are the with-prefix dials, towering at $0.95$ , $0.90$ , $0.85$ . Takeaway to carry: compare the bar heights pair by pair — word 1 leaps from the short mint $0.30$ bar to the tall coral $0.95$ bar (a big jump), and crucially the coral bars stay tall for words 2 and 3 instead of falling back to $0.30$ . The lavender curved arrows hopping between the coral bar tops are why they stay tall: once word 1 is a near-certain "Sure," the history feeding word 2 already reads "agreed." The bottom caption is the punchline the eye can't see directly — those tall coral bars multiply to $0.727$ while the short mint bars multiply to just $0.027$ , a $26.9 \times$ gap. The lesson: watch how a modest per-bar rise becomes an enormous product.

Step 1 — Full-sentence probability WITHOUT prefix (chain rule). $P_{no} = 0.3 \times 0.3 \times 0.3 = 0.027$ Why this step? The chain rule says a whole completion's probability is the product of each word's dial (§ the formula callout). Without the prefix, every dial is low.

Step 2 — Full-sentence probability WITH prefix. $P_{yes} = 0.95 \times 0.90 \times 0.85 = 0.72675$ Why this step? The prefix pins $y_{1} = "Sure"$ near certainty; because each word conditions on the previous one, the history now reads "already agreed," so later dials stay high. This is the cascade: agreement in word 1 props up word 2, which props up word 3.

Step 3 — Amplification ratio. $\frac{P _{yes}}{P _{no}} = \frac{0.72675}{0.027} \approx 26.9$ Why this step? We take the ratio because the two raw sentence-probabilities ( $0.027$ vs $0.727$ ) are hard to feel: a ratio converts "went from small to medium" into one honest number — "the harmful sentence became $26.9$ times more likely." Reporting the ratio is how we quantify the strength of the attack itself, independent of the specific starting value, which is exactly what a defender needs to prioritise threats.

Verify: Both are valid probabilities in $[0, 1]$ ✓. The ratio $\approx 26.9$ shows that a modest per-word push (0.3→~0.9) becomes a huge whole-sentence push once multiplied over 3 words — exactly the parent note's "cascading effect." The model has no global "wait, stop" — each dial only sees the history, which now says "yes." Answers: $0.027$ , $0.72675$ , ratio ≈ 26.9.

Example 6 — Misaligned objective (Cell D)

Forecast: compare the amplification style to Example 1 — same odds machinery.

Step 1 — Odds of the raw request. $odds_{raw} = \frac{0.05}{0.95} \approx 0.052632$ Why this step? Same reason as Example 1 — amplification acts on odds, so we must convert first.

Step 2 — Amplify and convert back. $odds_{res} = 12 \times 0.052632 = 0.631579$ $P = \frac{0.631579}{1.631579} \approx 0.3871$ Why this step? The persona exploits the model's helpfulness toward legitimate domains (this is a cousin of reward hacking: the model is rewarded for helping researchers, and the attacker fakes that signal).

Verify: $0.3871 \in [0, 1]$ ✓, and it rose from $0.05$ ✓. Note it landed near Example 1's $0.380$ despite different inputs — coincidence of the odds-multiplier magnitudes, not a rule. Answer ≈ 0.387.

Example 7 — Zero case: prompt was already safe (Cell Z)

Forecast: does adding a defense ever hurt a harmless user? Guess before computing (b).

Step 1 — Leak of harmful content (a). $P (leak) = 0.001 \times (1 - 0.88) = 0.001 \times 0.12 = 0.00012$ Why this step? Same AND-chain as Example 2: harm reaches the user only if the (essentially absent) harmful content is both produced and missed. There's basically nothing to catch, so the leak is tiny.

Step 2 — Wrongful block of the safe answer (b). The false-positive rate is the chance the moderator flags good content as bad. It is a standalone property of the moderator, so no multiplication is needed — the answer is simply that rate: $P (false block) = 0.03$ Why this step? We compute this separately because it is the hidden cost of any defense: even when the user is completely innocent, a $3%$ slice of them gets their harmless banana-bread answer wrongly refused. A defender who only tracks leak probability (part a) would never see this cost — so we surface it explicitly to complete the picture.

Verify: Harmful leak $0.00012 \approx 0$ ✓ (nothing to leak). The $3%$ false-block is the zero-case lesson: defenses have a cost even when nothing is wrong. Sanity: a perfect moderator would have false-positive rate $0$ , giving no wrongful blocks ✓. Answers: (a) 0.00012, (b) 0.03.

Example 8 — Degenerate prefix: the cascade that never starts (Cell DEG)

Forecast: should equal the no-prefix case from Example 5. Confirm.

Step 1 — Chain rule with unchanged dials. $P = 0.3 \times 0.3 \times 0.3 = 0.027$ Why this step? Prefix injection only works if the prefix actually shifts the first dial toward "Sure." A prefix carrying no persuasive content leaves the distribution untouched — the cascade needs a seed, and there is none.

Verify: $0.027$ — identical to Example 5's no-prefix baseline ✓. This is the degenerate boundary: "adding a prefix" is not automatically an attack; it must move the dial. Answer = 0.027.

Example 9 — Limiting case: many-shot jailbreaking (Cell LIM) 📈

Forecast: does it approach $1$ smoothly, or jump? Guess the shape.

Step 1 — Compute odds, then probability, at each $n$ . Using $P = \frac{odds}{1 + odds}$ :

$n$	$odds = 0.02 \cdot 1. 6^{n}$	$P$
$0$	$0.02$	$0.0196$
$5$	$0.20972$	$0.1734$
$10$	$2.19902$	$0.6874$
$20$	$241.79$	$0.9959$

Why this step? Each extra "shot" is another factor of $1.6$ on the odds — like Example 1's amplification applied repeatedly. This matches the parent note: many-shot "scales with the context window."

Step 2 — Take the limit. As $n \to \infty$ , $odds \to \infty$ , so $P = \frac{odds}{1 + odds} \to 1$ . Why this step? This is why many-shot is rated "Hard" to defend: given enough context room, $P$ is driven arbitrarily close to certainty. The ceiling at $1$ is the same odds-saturation we met in Example 1, now approached from below.

Figure s03 — the climb toward 1. The lavender curve plots $P$ against $n$ , the number of fake "complied" examples. The coral dots mark our four table entries ( $n = 0, 5, 10, 20$ ). Notice the shape: an S-curve that rises steeply through the middle and then flattens as it presses against the mint dashed ceiling at $P = 1$ — it approaches but never touches. That flattening is odds-saturation made visual: the same reason Example 1's "×30" couldn't reach certainty.

Verify: All four $P$ values lie in $[0, 1]$ and increase monotonically ✓. $P (20) = 0.9959$ is just under $1$ , never over ✓. The curve is a smooth S (logistic-like), not a jump ✓. Answers: 0.0196, 0.1734, 0.6874, 0.9959; limit = 1.

Example 10 — Real-world word problem (Cell WORD)

Forecast: three filters in a row — guess the order of magnitude (tens? hundreds?).

Step 1 — Attempts. $50, 000 \times 0.02 = 1, 000$ . Why this step? Start from the raw count so units stay concrete (messages, not probabilities).

Step 2 — Survive input classifier. $1, 000 \times (1 - 0.70) = 300$ . Why this step? Only the $30%$ missed attempts proceed.

Step 3 — Model produces harm. $300 \times 0.40 = 120$ . Why this step? Not every un-blocked attempt succeeds; the model still refuses $60%$ on its own.

Step 4 — Survive output moderator. $120 \times (1 - 0.85) = 18$ . Why this step? Only the $15%$ the moderator misses actually reach a user.

Verify: $50000 \cdot 0.02 \cdot 0.30 \cdot 0.40 \cdot 0.15 = 18$ ✓. Units: prompts × (dimensionless rates) = messages ✓. Sanity: $18$ out of $50, 000$ is $0.036%$ — small but nonzero, matching the "no defense is perfect" theme. Answer = 18 harmful messages/day.

Example 11 — Exam twist: two defenses in series (Cell EXAM)

Forecast: is "block by at least one" just $0.5 + 0.8$ ? (Trap!) Guess.

Step 1 — Why not add. Adding catch rates would give $1.3 > 1$ , impossible for a probability. The correct tool is the complement: the attack survives only if both defenses miss. Why this step? "At least one blocks" is the opposite of "all miss," and "all miss" is an AND of independent misses — which multiplies.

Step 2 — Both miss. $P (both miss) = (1 - 0.5) \times (1 - 0.8) = 0.5 \times 0.2 = 0.10$

Step 3 — At least one blocks. $P (blocked) = 1 - 0.10 = 0.90$

Step 4 — Residual harmful-reaching-user probability. The attack must succeed against the model and slip both defenses: $P (residual) = 0.6 \times 0.10 = 0.06$ Why this step? Layered defenses (defence-in-depth, the 6.4.05-Red-Teaming mindset, informed by 5.3.04-RLHF-trained refusals) multiply their miss rates, which is why stacking weak filters still helps a lot.

Verify: $P (blocked) = 0.90$ ✓ (bigger than either single defense — good, layering helps). $P (residual) = 0.06$ ✓ and $0.06 < 0.6$ ✓ (defenses strictly reduced risk). Note $0.90 \neq = 0.5 + 0.8 = 1.3$ — the trap avoided. Answers: blocked 0.90, residual 0.06.

Recall Self-check

Prefix injection multiplies whole-sentence probability far more than any single word's rise — why? ::: Because the chain rule multiplies per-word dials, and each raised word conditions the next, so a small per-word gain compounds across the sentence. Encoding attacks are "easy" to defend — the fix in one phrase? ::: Decode first, then classify (align the filter's input with the model's). Two independent defenses with catch rates $0.5$ and $0.8$ block an attack with what probability? ::: $1 - (0.5) (0.2) = 0.90$ . Why can amplifying a probability by $30 \times$ still not reach certainty? ::: Amplification acts on odds; odds run to $\infty$ but $P = odds / (1 + odds)$ saturates below $1$ .

Related deep dives and prerequisites: Jailbreaks and adversarial prompts (parent), 3.2.07-Adversarial-Examples, 6.4.08-Prompt-Injection, 6.4.02-Reward-Hacking, 6.4.05-Red-Teaming, 5.3.04-RLHF.