Visual walkthrough — Autonomous agent evaluation

We will earn every symbol before we use it. By the end you will see the number $0.9^{10}\approx 0.35$ instead of just reading it.

---

Step 1 — What is a "step" and what is "success"?

WHAT. Imagine the agent's job as a row of doorways. To finish the task it must walk through every doorway in order. Each doorway is one step — one action, like "click the search button" or "read the price".

WHY. Before any math, we need a picture of what we are counting. The parent note listed a flight-booking agent with 6 steps; here we draw the chain itself so "step" is a thing you can point at, not a word.

PICTURE. Below, each box is one step. A green tick means the agent got through that doorway; a red cross means it got stuck. If any doorway is a cross, the agent never reaches the goal at the far end.

---

Step 2 — Give each doorway a number: probability

WHAT. We attach a number to each doorway: the chance the agent clears that one doorway. We write it $p$.

WHY. We want to reason about reliability, not just eyeball it. A number lets us combine steps with arithmetic. We choose probability — a number between $0$ and $1$ — because "chance of clearing a doorway" is exactly what probability was invented to measure.

PICTURE. Same chain as before, but now each doorway wears a label: $p_i$ = "chance of clearing doorway $i$". The bar under each doorway shows how full that chance is (a nearly-full bar = very reliable).

A smart check: if $p_i=1$ the doorway is always open; if $p_i=0$ it is a wall.

---

Step 3 — Why we multiply the numbers (the AND rule)

WHAT. To succeed overall the agent must clear step 1 AND step 2 AND ... AND step $n$. When independent chances must all happen, we multiply them.

WHY multiply and not add? Adding is for "either this or that" (more ways to win → bigger number). But here every extra doorway is one more wall that could stop you, so the chance can only shrink. Multiplying two numbers that are each below $1$ gives something smaller — which is exactly the shrinking we expect.

PICTURE. Start with a full square = probability $1$ (certain). Each doorway keeps only a fraction $p$ of what is left. Two doorways at $0.9$: first keep $90\%$ of the square, then keep $90\%$ of that strip. The surviving area is the overall success chance.

---

Step 4 — All steps equally reliable: the power $p^n$

WHAT. Suppose every doorway is equally good, $p_i=p$ for all $i$. Then multiplying $p$ by itself $n$ times is raising it to the power $n$.

WHY. This is the clean, worst-truth-in-one-glance case. It turns a long product into a single expression $p^n$ we can plot and feel.

PICTURE. A curve of $p^n$ as $n$ grows, drawn for a few values of $p$. Notice how the $p=0.9$ curve, which starts near the top, sags dramatically as steps pile up — while $p=0.99$ holds up far longer.

---

Step 5 — Put the parent's numbers in: $0.9^{10}$

WHAT. Plug $p=0.9$ and $n=10$ into $p^n$.

WHY. This is the headline number from the parent note. We want to derive it, not quote it.

PICTURE. Ten bars, each shrinking the running product by a factor $0.9$. Watch the running total slide from $1.00$ down past $0.5$ and settle near $0.35$ — a "leaky bucket" losing $10\%$ at every pour.

$$P(\text{success})=0.9^{10}=\underset{10\text{ times}}{\underbrace{0.9\times 0.9\times \cdots \times 0.9}}\approx 0.349$$

---

Step 6 — Edge and degenerate cases (never leave a gap)

WHAT + WHY + PICTURE. We now sweep every corner so no reader hits an unshown scenario.

• $n=1$ (single step): $P=p^1=p$. One doorway, no chaining — the classic "static input → output" ML case. The compounding penalty vanishes.
• $p=1$ (perfect steps): $P=1^n=1$ for any $n$. A flawless agent never decays, no matter how long the task. The whole problem only exists because $p<1$.
• $p=0$ (a wall somewhere): $P=0$. One impossible step zeroes the entire product — this is the "single failed step blocks everything" fact made numeric.
• $n\to \infty$ (long-horizon task): since $0\le p<1$, $p^n\to 0$. Any imperfect agent, given a long enough task, eventually fails almost surely. Long-horizon agents need either near-perfect steps or error recovery (which breaks the independence assumption in the agent's favour — see Reinforcement Learning Basics).
• Uneven steps: if one $p_i$ is tiny (a bottleneck doorway), the product is dragged down almost entirely by it, whatever the others are. Fixing the weakest step buys the most success.

---

The one-picture summary

One figure, the whole story: a chain of doorways (Step 1) each labelled $p$ (Step 2), multiplied together via the AND rule (Step 3) into $p^n$ (Step 4), evaluated at $0.9^{10}\approx 0.35$ (Step 5), with the edge curves for $p=1$, $p=0$ and long $n$ (Step 6) framing it.

Recall Feynman retelling — say it back in plain words

Picture a hallway of doors. To finish, the agent must walk through every door. Each door opens for it about $9$ times out of $10$. That feels safe — but reliability doesn't add, it multiplies, because you need door 1 open and door 2 open and so on. Multiply ten $0.9$'s and you get about $0.35$: the agent finishes the whole hallway only a third of the time, even though it almost never trips on a single door. So a per-action score is a liar — measure the whole hallway, end to end. And if you ever want a long hallway to actually get finished, you can't just be "good" per door; you have to be almost perfect per door, or build agents that can back up and retry a door they missed.

Recall Quick self-checks

What operation combines independent per-step chances, and why? ::: Multiplication, because success needs step 1 AND step 2 AND ... — all must happen. Compute $0.9^{10}$ to two decimals. ::: About $0.35$. If $p=1$, what is $p^n$ for any $n$? ::: $1$ — a perfect agent never decays. To hit $90\%$ task success over $10$ steps, what per-step $p$ do you need? ::: $0.9^{1/10}\approx 0.99$. Why does per-action accuracy overstate an agent's real ability? ::: Because task success is the product over steps, which shrinks fast; accuracy hides the compounding.

---

Related: Benchmark Design · Human Evaluation Protocols · Tool Use in Language Models · Multi-Agent Systems · A/B Testing · Safety Alignment · LM Prompting Strategies