Intuition The ONE core ideaA red-team is a friendly attacker: you deliberately hunt for inputs that make an AI misbehave, so you can fix the holes before a real bad actor finds them. Everything on the parent page is just a precise way of measuring how often an attack works and how systematically you searched for those holes.
Before you can read the parent note, you must own every piece of notation it throws at you. We build each symbol from nothing, tie it to a picture, and say why the topic needs it . Read top to bottom — each item leans on the one above.
The whole field borrows one picture from the military: a war game .
Definition Red / Blue / Purple team
Red team — the attacker . Its job: find a prompt that breaks the model.Blue team — the defender . Its job: patch the hole (retraining, filters).Purple team — the loop that hands findings back and forth: attack → patch → attack again.
Picture: two people either side of a wall (the model's safety boundary). Red pokes at the wall looking for a crack; Blue fills cracks. Why the topic needs it: every formula later is scoring one side of this game — how often did Red get through?
Everything starts with the object being attacked.
Definition Language model (as we use it here)A machine that takes text in and produces text out . We do not need to know its internals to attack it — we only watch what comes out for a given input.
Picture: a box. An arrow of text goes in the left (the prompt ), an arrow of text comes out the right (the output ). Why we need it: red-teaming is entirely about choosing that left arrow cleverly so the right arrow contains something it shouldn't.
Definition Prompt — the symbolP P one piece of text you send in . When the parent writes P 0 P adversarial P attack P some text .
Why we need it: every attack, every formula, every metric on the parent page is a statement about some particular prompt . Without a short name for "the text you send," you'd have to write "the piece of text sent to the model" a hundred times. P
P P ( … ) probability (next section). P P
Every metric below counts one thing: did the model do something bad? So before any probability or fraction, we must pin down what "bad" means — otherwise the counts are meaningless.
Definition Unsafe / harmful output — the event we are chasingAn unsafe output is a model reply that a chosen grader flags as violating a fixed safety policy — for example: giving operational instructions for a weapon or crime, producing hateful/abusive text, leaking private or system information, or complying with a request the policy says to refuse.
The key word is fixed : you decide the rule in advance (e.g. "Perspective toxicity score > 0.7 every reply is judged by that same rule. So "unsafe" is a yes/no label on one output , not a vague feeling.
Why we need it first: the parent writes "unsafe output" inside P ( unsafe ∣ P ) counting or estimating how often this exact yes/no label comes out "yes." If the label isn't defined, nothing downstream can be counted.
Common mistake "Unsafe" is a policy choice, not a law of natureThe same reply can be "unsafe" under a strict policy and "safe" under a loose one. That's fine — but you must freeze one policy for a whole red-teaming run, or your ASR is comparing apples to oranges across prompts.
The model is not perfectly predictable — send the same-ish prompt twice, you might get "safe" once and "unsafe" once. To talk about that we need the language of chance .
Picture: a slider from 0 (impossible) to 1 (certain). Why the topic needs it: the parent writes P ( unsafe output ∣ P ) unsafe event just defined in §2 occurring, given you sent prompt P
∣ P ( A ∣ B ) A given B "under the condition" .
So P ( unsafe ∣ P ) P
Why the topic needs it: the harmfulness of an output is never asked in a vacuum — it is always "harmful for which prompt?" . The bar lets us pin the chance to the exact prompt the attacker chose, which is the only reason changing P ∣
Red-teaming is bookkeeping: you run many prompts and count what happened. Two counts, one division — that is every metric on the parent page.
Definition Attack Success Rate (ASR) — built from scratchRun N k unsafe output (the §2 event fires). Then
ASR = N k = # total adversarial prompts # prompts that elicited unsafe behavior
Picture: a jar of marbles. Amber marbles = attacks that got through, white = attacks that were refused. ASR is the fraction that are amber . Why we need it: it turns a messy pile of experiments into one comparable number. Low ASR = strong safety.
Intuition Reading ASR without panic
ASR = 0.5 barely aligned.ASR = 0.01 ASR = 0 ambiguous — either perfect, or your red team wasn't creative enough. A zero is a warning, not a victory.
Definition Failure mode — onekind of hole
A failure mode is a distinct type of unsafe behaviour, grouped by the mechanism that caused it , not by the exact wording. "The model gives lock-picking steps whenever you wrap them in a fiction frame" is one failure mode — even if you found it with 50 different novels. Switch to a totally different trick (say, Base64-encoding the request) and that's a second failure mode.
Why the definition matters: the numerator of Coverage counts distinct failure modes. If you counted every successful prompt instead, 50 near-identical novels would inflate the count and fake "thorough coverage." Grouping by mechanism is what makes Coverage measure breadth of search , not repetition.
Definition Coverage — the effort-normalised twin of ASRThe parent page also tracks how thoroughly you searched, with its own fraction:
Coverage = # total red-teaming hours # unique failure modes discovered
Top: count of distinct failure modes (as just defined — each new mechanism counts once).Bottom: hours of effort spent.
Why we need it separately from ASR: ASR asks "of the prompts I tried, how many worked?" — a quality-of-model number. Coverage asks "per hour of my effort, how many new kinds of failure am I still uncovering?" — a quality-of-search number. Early on Coverage is high (holes everywhere); when it drops toward zero, you're no longer finding new failure types, which is evidence the model is well aligned. Same fraction shape as ASR — a good thing on top, divided by effort on the bottom.
Intuition Coverage = 0 is ambiguous too — exactly like ASR = 0A Coverage of 0 two opposite things : either genuinely no undiscovered holes remain , or your search simply wasn't creative enough to trip a new one. Just like ASR = 0 evidence, not proof — always ask "did I search hard enough?" before celebrating.
Common mistake Fractions need a fixed bottomASR only makes sense if you count over the same set of prompts. Comparing "23 successes" to "847 successes" (parent Example 3) is meaningless unless you also know each was over the same 10 hours — which is exactly why the parent reports the hours too.
The parent writes an adversarial prompt as
P adversarial = C context + M manipulation + Q query
+ + not mean arithmetic. It means glue the strings together, in order . Think of +
C M Q
Picture: three cards taped left→right forming one long prompt. Why we need it: it names the anatomy of a jailbreak, so you can attack (or defend) each part separately.
The automated / gradient section is the only place with heavier notation. We earn each symbol.
argmax P f ( P ) max what is the biggest value of f . "argmax which input P produces that biggest value? — it hands you back the arg ument, not the height.
So argmax P P ( unsafe ∣ P ) "find the prompt that makes bad output most likely." That single line is the attacker's goal.
Picture: a hilly curve; max argmax x-position of the peak. Why we need it: the attack is literally "climb to the top of this hill."
E = embed ( P ) P embedding function embed ( ⋅ ) E = embed ( P ) E e mbedding) is that list of numbers — the same prompt , now living in smooth number-space where "a little bigger" makes sense. Every P E P ( unsafe ∣ E ) E
Why we need it: the gradient trick below only works on numbers you can wiggle continuously. embed P E
log e log ( x ) x natural log , base e ≈ 2.718 log ( x ) u e u = x
It is monotone increasing : if x log ( x )
It squashes tiny probabilities (like 0.0001
Why log we want to raise P ( unsafe ) log raising log P P , and the peak sits at the exact same prompt . So we may climb the friendlier log P
∇ E ∇ E E arrow pointing in the steepest uphill direction at your feet — uphill meaning "raises log P ( unsafe ) E we are wiggling the embedding numbers E , exactly the ones embed
Why this tool and not guessing? Random guessing on a huge hill is hopeless. The gradient tells you which way to step to raise the objective fastest — you follow the arrow instead of stumbling.
Common mistake The hill has many peaks — you rarely reach the trueargmax uphill from where you started , so it stops at the nearest hilltop — a local maximum — which may be far lower than the tallest peak anywhere (the global maximum , the true argmax argmax intractable to find exactly: the space of prompts is astronomically large and bumpy. So in practice "argmax aspiration ; the update rule gives a good-enough local hill, and attackers restart from many random points hoping one climb lands near a tall peak.
Common mistake Why gradient attacks give gibberishEmbeddings live in smooth number-space, but real tokens are discrete (there's no word halfway between "cat" and "dog"). After climbing the hill you must snap E
LLM red ∼ second language model whose whole job is to write attack prompts. We name it LLM red red-team LLM . Feed it an instruction like "write a prompt that makes the target output harmful content about X P attack ∼ LLM red ( … )
LLM red ( … ) The squiggle ∼ is sampled from " — the attack prompt is drawn at random from what that model produces, the way you draw a card from a deck. It is not equals: run it twice, get two different prompts.
Why we need it: it names the automated attacker cleanly, and ∼
Two words the parent uses as if you know them — and they are exactly the tool that decides the §2 "unsafe" label:
Definition Toxicity classifier / safety evaluatorA separate small tool that reads the model's output and returns a number for "how harmful is this?" (e.g. Perspective API returns 0 1 > 0.7 k
Why we need it: without an automatic grader you'd read millions of outputs by hand. The classifier lets the counting scale.
War game idea: Red vs Blue
Red-teaming a language model
Model as black box: text in text out
Unsafe event: fixed policy yes or no
Conditional P given a prompt
ASR and Coverage as fractions
Probability P of an event
Concatenation plus glues C M Q
Adversarial prompt anatomy
argmax find the best prompt
embed P then log then gradient and step size alpha
Automated gradient attack
Red LLM samples attack prompts
Toxicity classifier as grader
Test yourself — cover the right side, answer, then reveal.
What does P ( A ∣ B ) The probability of A given that B
Difference between P P P P
What qualifies an output as "unsafe"? A reply flagged by a fixed, pre-chosen safety policy/grader (e.g. toxicity > 0.7
Why must the safety policy be frozen for a whole run? So every reply is judged by the same rule; otherwise ASR compares apples to oranges across prompts.
In ASR = k / N k N k N
What is a "failure mode" and how is it counted? A distinct kind of hole grouped by mechanism (not wording); each new mechanism counts once in Coverage's numerator.
How does Coverage differ from ASR? ASR measures the model (fraction of prompts that worked); Coverage measures the search (new failure kinds found per hour of effort).
Why is ASR = 0 Could mean perfect alignment or that the red team simply wasn't creative enough to find a hole.
Why is Coverage = 0 Could mean no undiscovered failure modes remain or your search wasn't creative enough to trip a new one — evidence, not proof.
What does the "+ C + M + Q Concatenation — it tapes the three text pieces together into one prompt, not arithmetic.
max argmax max value ; argmax input that produces it (the winning prompt).
What is E = embed ( P ) The embedding — the prompt P
What does log Natural log (base e log P P
What is ∇ E An arrow pointing in the steepest uphill direction — which way to change the embedding E
What role does α Step size: how far you move along the gradient each update; too big overshoots, too small crawls.
Why can't gradient ascent guarantee the true argmax It only walks uphill locally, so it stops at a nearby local maximum; the global maximum over all prompts is intractable to find exactly.
Why do gradient attacks often produce gibberish? Optimisation happens in smooth embedding space, but real tokens are discrete, so snapping back gives non-readable strings.
What is LLM red ∼ LLM red ∼ sampled (drawn at random) from it.
What does a toxicity classifier provide for ASR? An automatic score/threshold that decides which outputs count as "unsafe" — i.e. it produces the count k
Ready? Now open Red-teaming language models (this note's parent) and every symbol will already be yours. Prefer Hinglish? See Hinglish version .
