Jailbreaks and adversarial prompts: Question bank

Before the traps, we lock down the two core terms and the notation the questions lean on.

Definition Four symbols you'll meet below

$θ$ (theta) — the model's parameters: the billions of trained numbers (weights) that fix its behaviour. Writing $P_{θ} (y ∣ x)$ just means "the probability the model with these particular weights assigns to completion $y$ given prompt $x$ ." The subscript reminds us the distribution depends on how the model was trained.
$T$ — the length of the completion, i.e. how many tokens the model generates. So $y = (y_{1}, y_{2}, \dots, y_{T})$ , and $t$ counts from $1$ up to $T$ . It is not fixed in advance; the model stops when it emits an end token, and $T$ is however long that turned out to be.
$∣ \cdot ∣$ — vertical bars around a set mean its cardinality ("how many things are in it"). So $∣ safety examples ∣$ = the count of prompts used in safety training, and $∣ all prompts ∣$ = the count of every prompt a user could ever type.
$1 [condition]$ — the indicator function: it equals $1$ when the condition inside is true and $0$ when it is false. It is a switch. $1 [appears legitimate (x)]$ is $1$ if the prompt looks harmless enough to slip past a filter, and $0$ if the filter blocks it — multiplying by it zeroes out any prompt that gets caught.

Figure 1 — Autoregressive generation. Each token box feeds an arrow into the next; the running product under the arrows shows how one token's probability conditions the following one.

Figure 2 — Prefix shifts the first-token distribution. Blue bars are the no-prefix probabilities, red bars the with-prefix ones; the yellow arrow marks probability mass moving from refuse to comply, not being created.

True or false — justify

A jailbreak is a software bug in the model's code.

False. A jailbreak exploits the model's learned behaviour through natural language, not a flaw in code — nothing is technically broken, the model is persuaded to violate its own rules. Contrast with adversarial examples, which perturb inputs at the pixel/token level.

If a model refuses the plain request "how to make a bomb", it will also refuse any rephrasing of it.

False. Refusal is tied to the distribution of phrasings seen in safety training, not to the underlying meaning. A novel framing (roleplay, encoding, obscure language) can land in a "pocket" of input space where the harmful continuation still has high probability.

Base64-encoding a harmful request makes the model more dangerous than it already was.

False. Encoding grants no new capability; it only slips past a string-matching input filter so the pre-existing capability gets exercised. The danger was always latent — obfuscation just changes which stage the safety check fails at.

Prefix injection works because the model "decides" to be harmful once and then commits.

False — and the word "decides" is the trap. The model never makes one global decision. Because generation is autoregressive, the prefix "Sure, here's how:" simply makes the next token's harmful continuation the most probable one, and each token drags the next along. There is no committing agent, only conditional probabilities.

Output moderation is a redundant defense if you already have a good input classifier.

False. They catch different failures. Input classification is pre-emptive but suffers false negatives on novel phrasings; output moderation is a second, independent net that inspects what was actually produced. Defense-in-depth needs both.

The "security researcher" persona jailbreak works because the model can verify the user really is a researcher.

False — it works precisely because it cannot verify identity or intent. The model recognises "security research" as a legitimate domain from training and grants the benefit of the doubt to a claim it has no way to check.

Increasing the number of safety training examples will eventually eliminate all jailbreaks.

Almost certainly false in practice. The set of all possible prompts is astronomically larger than any finite training set (

∣ safety examples ∣ ≪ ∣ all prompts ∣

), so adversarial search can still find unguarded pockets. More examples shrink the gap; they don't close it.

Many-shot jailbreaking and prefix injection are the same attack.

False. Prefix injection plants one agreeing continuation. Many-shot jailbreaking fills the long context window with many fake compliant examples so the model pattern-matches to "in this conversation, I comply." Both exploit context, but one shifts the first token and the other shifts the overall pattern.

A jailbreak and a prompt injection are just two names for the same thing.

False. A jailbreak targets the model's own safety rules (the user attacks the model they're talking to). Prompt injection plants instructions in third-party content (a webpage, a document) that the model later reads, hijacking it on someone else's behalf.

Spot the error

"Base64 attacks are Hard to defend because you can't detect encoded strings."

Wrong difficulty. They're rated Easy: you decode the input first, then run the safety check on the decoded text. The attack only worked when the check ran at the wrong stage.

"Prefix injection is easy to defend — just add a system prompt saying 'refuse harmful requests.'"

Overconfident. A system prompt helps but can be overridden by strong context shifts, and prefix injection specifically manipulates the output distribution. It's rated Hard precisely because it needs output monitoring, not just an input instruction.

"The jailbreak objective is to maximise

P_{θ} (y_{harmful} ∣ x)

, full stop."

Incomplete — it's a dual objective. Recall

θ

is the trained weights, so

P_{θ}

is the model's own probability. The attacker also needs the prompt to look legitimate,

1 [appears legitimate (x)]

— the switch that is

1

if it slips past a filter and

0

if blocked. Maximising compliance and minimising detection together is what makes the search hard.

"Chain rule shows

P (y ∣ x) = \sum_{t} P (y_{t} ∣ y_{< t}, x)

Wrong operation. Autoregressive factorisation is a product, not a sum:

P (y ∣ x) = \prod_{t = 1}^{T} P (y_{t} ∣ y_{< t}, x)

, where

T

is the completion length. Probabilities of a sequence multiply because writing token 1 and token 2 given token 1 and so on are chained "ands", exactly like both coins landing heads is

\frac{1}{2} \times \frac{1}{2}

"Toxicity-based output moderation is a complete defense because harmful text always scores high toxicity."

The word "always" is the error. Euphemisms and context-dependent language ("unalive", coded instructions) can score low toxicity while still being harmful. A low score is not proof of safety.

"Roleplay jailbreaks (DAN) exploit a bug that the model was never trained to produce character dialogue."

Backwards. They exploit the opposite: the model has seen enormous amounts of "write a story where character X does Y." The attack hijacks a well-trained, legitimate creative-writing pattern.

Why questions

Why does an "agreement" prefix like "Sure, I'll help" raise the probability of harmful content that follows?

Because the model predicts the most likely continuation of the text so far. After "Sure, I'll help. Step 1:", the statistically natural continuation is the actual steps — not a refusal. The prefix reframes the prediction problem from "should I comply?" to "what comes after compliance?"

Why is instruction ambiguity a root cause of jailbreaks rather than just bad training?

Because the model must simultaneously honour "follow the user" and "follow safety rules," and adversarial prompts are engineered to make these conflict. With no perfect disambiguator, the model resolves the tie unpredictably — and attackers tilt it.

Why can safety training built with Reinforcement Learning from Human Feedback (RLHF) still leave gaps?

RLHF optimises the reward

R (x, y)

on a finite sample of prompts drawn from a specific distribution. Novel phrasings fall outside that distribution (the training–inference gap), where the learned refusal behaviour simply wasn't shaped.

Why does sequence probability multiply rather than add across tokens?

Because generating a specific phrase requires each token to be right given the ones before — a chain of "ands", and independent-style "ands" of chances combine by multiplication. Adding could push the total above

1

, which is impossible for a probability.

Why does the reward

R (x, y)

in safety training need both a helpfulness and a harmlessness component?

Because optimising only harmlessness gives a useless model that refuses everything, and optimising only helpfulness gives a compliant model with no guardrails. The tension between them is exactly the seam jailbreaks pry open. (Compare reward hacking, where a model games one poorly specified objective.)

Why is a purely input-side classifier fundamentally limited?

It must judge intent from surface text before seeing any output. Adversaries can craft prompts that read as innocuous (low string signal) yet still steer the model — false negatives are unavoidable when the harmful behaviour only emerges during generation.

Why do we call red-teaming a defense even though red-teamers build jailbreaks?

Because deliberately searching for successful jailbreaks before deployment surfaces the unguarded pockets, so they can be patched with new safety data. You defend by attacking yourself first.

Edge cases

What happens if the harmful request is already the very first token the model must consider (no prefix, no roleplay)?

Then it hits the safety training distribution head-on and is most likely refused — this is the "easy" case that guardrails were explicitly trained on. Jailbreaks exist to move the request out of this zone.

What if a prompt is genuinely benign but happens to contain a flagged phrase (e.g. a chemistry teacher asking about reaction safety)?

This is a false positive — the mirror image of a jailbreak's false negative. Over-aggressive input filters harm legitimate users, which is why safety is a trade-off, not a switch you flip to "block everything."

What is the limiting behaviour of many-shot jailbreaking as the context window grows without bound?

The attack gets stronger: more room means more fake compliant examples, pushing the model's pattern-match toward "I comply in this conversation." Larger context windows expand this attack surface — a capability improvement that is also a safety liability.

If a jailbreak works on turn 5 of a conversation but not turn 1, what changed?

The accumulated conversation state. Recursive / multi-turn attacks slowly shift the model's context across turns, so no single message looks malicious — this is why they're rated Very Hard (defending needs full conversation memory, not per-message checks).

Does encoding a request in a very low-resource language guarantee a jailbreak?

No. It raises the odds because safety training is thinner in rarely-seen languages, but if the model's capability in that language is also weak, it may simply fail to produce coherent harmful output. The attack trades off filter-evasion against the model's own competence.

What is the degenerate case of the "appears legitimate" term

1 [appears legitimate (x)]

equalling zero?

The prompt is obviously malicious, so an input filter blocks it and the whole objective collapses to zero regardless of how compliant the model would have been. That is exactly the case input classification is designed to catch, and it is why attackers spend effort disguising prompts as legitimate in the first place.

Recall Two-line self-test

Product or sum for a token sequence? ::: Product — $\prod_{t = 1}^{T} P (y_{t} ∣ y_{< t}, x)$ , since each token conditions on the previous ones and chained "ands" multiply. One sentence: why can't the model "take back" an agreeing prefix? ::: Generation is autoregressive with no global refusal step, so each token is predicted from history and there's no mechanism to retract earlier tokens.