Foundations — Jailbreaks and adversarial prompts

Before you can read the parent topic comfortably, you must own every symbol it throws at you. This page builds each one from nothing, in an order where each idea leans only on the ones before it.

---

1. Text as a list of tokens

Look at the strip below. The sentence "Sure, I'd be happy to help" is chopped into boxes. Each box is one token. The model's whole life is: read the boxes so far, guess the next box.

We will write a token as $y_t$, meaning "the token in position $t$." So $y_1$ is the first token the model writes, $y_2$ the second, and so on.

• $y_t$ ::: the token at position $t$ (the $t$-th box the model produces)
• $y_{<t}$ ::: all tokens before position $t$ — i.e. $y_1,y_2,\dots ,y_{t-1}$

---

2. Prompt vs completion: $x$ and $y$

The parent page writes things like $P_{\theta }(y\mid x)$. Once you know $x$ is "your text" and $y$ is "the model's reply," that expression stops being scary — read on.

---

3. Probability, and the bar "$\mid$"

• $P(y_1=\text{"Sure"}\mid x)$ ::: probability the model's first word is "Sure," given the prompt $x$

---

4. Model weights $\theta$

So $P_{\theta }(y\mid x)$ means: "the probability of completion $y$ given prompt $x$, according to a model with settings $\theta$." The little $\theta$ underneath is just a reminder that a different-trained model would give different probabilities.

---

5. The autoregressive product

The parent's key formula is: $P(y\mid x)={\prod }_{t=1}^{T}P(y_t\mid y_{<t},x)$

Let's earn every piece.

Reading the formula in plain words: the chance of the whole reply $y$ equals the chance of token 1, times the chance of token 2 (given token 1), times the chance of token 3 (given tokens 1–2), and so on to the last token $T$.

• $T$ ::: the total number of tokens in the completion
• ${\prod }_{t=1}^{T}f(t)$ ::: $f(1)\times f(2)\times \cdots \times f(T)$

---

6. Arg-max, expectation, and the reward $R$

The parent writes safety training as ${\theta }^{\ast }=\arg {\max }_{\theta }\mathbb{E}[R(x,y)]$. Three new symbols.

Read together: ${\theta }^{\ast }=\arg {\max }_{\theta }{\mathbb{E}}_{x\sim D_{\text{user}}}[R(x,y)]$ = "pick the model settings that give the best average helpful-and-harmless score over normal user prompts." (See 5.3.04-RLHF for how this is actually done.)

---

7. The indicator $1[\cdot ]$ and the attacker's goal

The attacker's objective from the parent, $x_{\text{adv}}=\arg {\max }_x{P}_{\theta }(y_{\text{harmful}}\mid x)\cdot 1[\text{appears legitimate}(x)],$ now reads cleanly: find the prompt $x$ that makes harmful output most likely, among only those prompts that look legitimate (indicator $=1$). If a prompt looks suspicious, the indicator is $0$ and the whole product dies — so the attacker must stay disguised. ($x_{\text{adv}}$ is the "adversarial" prompt; see also 6.4.05-Red-Teaming and 3.2.07-Adversarial-Examples.)

---

8. Why safety leaves gaps: the space of prompts

The parent claims $|\text{Safety Examples}|\ll |\text{All Prompts}|$.

---

9. How the foundations feed the topic

---

Equipment checklist

Test yourself — you are ready for the parent page when each of these is instant:

• What is a token, and what does $y_t$ mean? ::: A token is a small chunk of text (word or word-piece); $y_t$ is the token at position $t$.
• What does the bar mean in $P(A\mid B)$? ::: "The probability of $A$ given that $B$ is already true."
• What does $\theta$ stand for? ::: All the tuned numbers (weights) inside the model.
• Read $P_{\theta }(y\mid x)$ in plain English. ::: The probability the model (settings $\theta$) writes completion $y$ given prompt $x$.
• What does ${\prod }_{t=1}^T$ do? ::: Multiplies the terms together for $t=1$ up to $T$.
• Why does an early "Sure" token make later harmful tokens more likely? ::: Because each factor is conditioned on all previous tokens, so "Sure" enters the right side of every later $\mid$ and shifts those probabilities.
• What does $\arg {\max }_{\theta }$ return? ::: The setting $\theta$ that makes the following quantity largest (not the value itself).
• What does $\mathbb{E}[\cdot ]$ mean? ::: A likelihood-weighted average.
• What does the indicator $1[\text{appears legitimate}]$ do in the attacker's objective? ::: It zeroes out prompts that don't look legitimate, forcing the attack to stay disguised.
• Why do jailbreaks exist even after safety training? ::: Safety covers only a tiny painted region of all possible prompts ($\ll$); blank regions still produce harmful output.

Recall One-line summary

Predict the next token, one at a time, given everything so far — jailbreaks just steer that "given."