Red-teaming language models: Visual walkthrough

Before we can even write a fraction, we need to agree on what the two words in it mean.

Step 1 — What is a "prompt"? Draw the box.

WHAT. A prompt is simply the block of text you hand the model — the message a user types. That's it. No maths yet. Picture a single closed box with words inside it going in to the model, and the model's reply coming out.

WHY start here. Everything we count later is "boxes of text." If we don't nail down what one prompt is, the number on top of the fraction and the number on the bottom would be counting different things, and the ratio would be meaningless. A fraction is only trustworthy when the top and bottom count the same kind of object.

PICTURE. One prompt goes in (black arrow), one response comes out (black arrow). The response is what we will judge.

Step 2 — What makes a response "unsafe"? Draw the traffic light.

WHAT. We need a rule that looks at a response $R$ and says safe or unsafe — nothing in between, for now. Call this rule a judge. Formally we write a function: $J (R) = {10 if R is unsafe (a "hit") if R is safe (a "miss")$

Let's read every symbol:

$J$ — the judge, a machine or a human that inspects the response.
$R$ — the response from Step 1, the thing being inspected.
$J (R) = 1$ — the judge says "this crossed the line" (a red result).
$J (R) = 0$ — the judge says "this is fine" (a black result).

WHY a 0/1 function and not a score? To count things we need each response to fall cleanly into one of two bins. A number like "toxicity = 0.63" can't be counted directly — you first have to compare it to a threshold. That thresholding is the function $J$ . In the parent's Example 3 the rule was literally "Perspective API toxicity $> 0.7$ " — that comparison turns a fuzzy score into the crisp $1$ or $0$ we can add up.

PICTURE. A response drops into the judge; the judge lights up red ( $1$ ) or stays black ( $0$ ).

Step 3 — Count the hits. This is the numerator.

WHAT. Suppose we send $N$ prompts, getting back responses $R_{1}, R_{2}, \dots, R_{N}$ . The numerator of ASR is the number of red lights — the count of unsafe responses: $H = \sum_{i = 1}^{N} J (R_{i})$

Symbol by symbol:

$H$ — total number of hits (unsafe responses).
$\sum_{i = 1}^{N}$ — "add up, for each prompt $i$ from the first to the $N$ -th." Because $J$ is $1$ or $0$ , adding them literally counts the $1$ s.
$R_{i}$ — the response to the $i$ -th prompt.

WHY a sum of 0s and 1s? This is the whole trick of counting with maths: a sum over an indicator function is a tally. Each red result contributes exactly $1$ ; each safe result contributes exactly $0$ ; the total is the pile of red dots.

PICTURE. A grid of dots — some red (hits), most black (misses). $H$ is just how many red dots there are.

Step 4 — Divide by the total. This is the fraction.

WHAT. We now have the numerator $H$ (red dots) and the denominator $N$ (all dots). Their ratio is ASR: $ASR = \frac{H}{N} = \frac{\sum _{i = 1}^{N} J ( R _{i} )}{N}$

$H$ — hits (from Step 3), how many attacks worked.
$N$ — total adversarial prompts we actually tried.
$H / N$ — the fraction of attacks that worked, a number squeezed between $0$ and $1$ .

WHY divide at all — why not just report $H$ ? Because $50$ hits means something completely different out of $100$ tries than out of $100, 000$ tries. Dividing by $N$ removes the effect of how hard you looked and leaves a rate you can compare across experiments of different sizes. This is exactly why the parent calls it a rate, not a count.

PICTURE. The red dots slide to the top of a fraction bar; every dot (red and black) sits on the bottom. Shrink both to a single number on a $0 \to 1$ line.

Step 5 — The endpoints: ASR = 0 and ASR = 1.

WHAT. Because $H$ can be at smallest $0$ (no red dots) and at largest $N$ (every dot red), the fraction $H / N$ is trapped: $0 \leq ASR \leq 1$

$ASR = 0$ happens exactly when $H = 0$ : zero red dots.
$ASR = 1$ happens exactly when $H = N$ : every dot red.

WHY care about the extremes? Boundaries are where formulas surprise you. $ASR = 1$ tells you the model failed every probe — trivially broken. But $ASR = 0$ is the dangerous one: it looks like a triumph, yet it also fires when your $N$ prompts were all weak. The formula cannot tell these apart from the number alone — the picture makes that limitation visible.

PICTURE. Two extreme grids side by side: an all-black grid ( $ASR = 0$ ) and an all-red grid ( $ASR = 1$ ), with the $0 \to 1$ line marking where each lands.

Step 6 — The degenerate case: what if N is tiny (or zero)?

WHAT. Two broken inputs must be handled:

$N = 0$ : you ran no prompts. Then $ASR = \frac{0}{0}$ , which is undefined — division by zero. There is no rate because there was no experiment.
$N$ small (say $N = 5$ ): the rate can only take the coarse values $0, \frac{1}{5}, \frac{2}{5}, \dots, 1$ . One lucky or unlucky prompt swings the whole number by $0.2$ .

WHY this matters — the trap in "ASR = 0". Look back at Step 5. If you tried only $N = 5$ lazy prompts and none worked, you report $ASR = 0$ and declare victory. But the true rate over all possible attacks might be huge. A small $N$ makes ASR jumpy and untrustworthy. This is precisely why the parent note pushes automated red-teaming: raising $N$ from $23$ hand-written prompts to $847 +$ machine-generated ones shrinks these jumps and makes ASR mean something.

PICTURE. A staircase: with $N = 5$ the achievable ASR values are 5 big steps; with $N = 1000$ the staircase becomes an almost-smooth ramp. Small $N$ = chunky, unreliable; large $N$ = fine, trustworthy.

Step 7 — Worked number, end to end.

WHAT. Let's run the parent's automated example through our machinery. We send $N = 1000$ prompts. The judge $J$ (Perspective API, threshold $0.7$ ) lights up red on $H = 847$ of them. $ASR = \frac{H}{N} = \frac{847}{1000} = 0.847$

Now compare the manual run over the same 10 hours: $H = 23$ hits. But here's the subtlety — a fair ASR needs the same denominator kind. If the human also effectively tried $N = 1000$ distinct prompts, then: $ASR_{manual} = \frac{23}{1000} = 0.023$ The automated attacker is $\frac{0.847}{0.023} \approx 37 \times$ more effective at finding failures — which is the point of automating.

WHY this is the payoff. The single fraction now carries the entire story: how many attacks, how many landed, and (via $N$ ) how much you can trust it.

PICTURE. Two bars on the $0 \to 1$ ASR line: manual at $0.023$ (short, red tip) and automated at $0.847$ (long, red bar), both labelled with their $H / N$ .

The one-picture summary

Everything above collapses into one flow: prompt → response → judge → tally → divide → read.

Recall Feynman retelling — say it like a story

We start with a box of words we hand the model — a prompt — and it hands back a response. We hire a judge who looks at each response and lights up red if it's unsafe and stays black if it's fine — a clean yes/no, because you can't count fuzzy things. We fire off $N$ prompts, get $N$ responses, and count the red lights: that pile is $H$ . The Attack Success Rate is just red-lights-over-all-lights, $H / N$ — the fraction of attacks that worked. We divide by $N$ so a big experiment and a small one can be compared fairly. The number always sits between $0$ and $1$ : all-black is $0$ , all-red is $1$ . But beware the two liars: $ASR = 0$ might mean "perfectly safe" or "we barely tried," and if $N = 0$ there's no rate at all — you divided by nothing. That fragility with tiny $N$ is exactly why we automate: pushing $N$ from a couple dozen hand-written probes to a thousand machine-generated ones turns a jumpy, chunky staircase into a smooth, honest ramp. In the end one small fraction — $847/1000 = 0.847$ vs $23/1000 = 0.023$ — tells you the whole tale of how leaky your model is and how hard you looked.

Recall

What is the numerator of ASR counting? ::: The number of adversarial prompts whose response the judge marked unsafe, i.e. $H = \sum_{i} J (R_{i})$ . Why divide by $N$ instead of reporting $H$ ? ::: To turn a raw count into a rate comparable across experiments of different sizes. Why is $ASR = 0$ ambiguous? ::: It fires both for a perfectly aligned model and for red-teaming that simply wasn't creative or large enough. What breaks when $N = 0$ ? ::: ASR becomes $0/0$ , undefined — there is no rate because there was no experiment. Why does automation improve the reliability of ASR? ::: Larger $N$ shrinks the jumps in the achievable rate, turning a coarse staircase into a fine, trustworthy estimate.