Worked examples — Autonomous agent evaluation

This is the hands-on companion to Autonomous Agent Evaluation. The parent note built the formulas; here we exercise every one of them on a matrix of cases — every sign, every degenerate input, every limiting value — so that when you meet one in an exam or a real eval run, you have already seen its twin.

Before any symbol appears, we re-anchor it. Nothing below assumes you memorised the parent.

---

The scenario matrix

Think of every evaluation problem as living in a grid. The rows are the quantities the parent taught; the columns are the awkward situations that break naive intuition — a zero here, a limit there, a sign flip. Our job is to land at least one worked example in every cell that matters.

Cell	Quantity under test	The awkward situation	Example
A	Compounding success (product of step rates)	many steps, per-step near 1	Ex 1
B	Compounding success	the degenerate step, rate $=0$	Ex 1
C	Success rate + confidence interval	small $n$ (wide bars) vs large $n$	Ex 2
D	Success rate CI	boundary cases $\hat{p}=1$ and $\hat{p}=0$ (zero variance)	Ex 3
E	Discounted reward $R=\sum {\gamma }^{t-1}{r}_t$	ordinary $\gamma \in (0,1)$	Ex 4
F	Discounted reward	limiting values $\gamma =0$ and $\gamma \to 1$; infinite horizon	Ex 4
G	Efficiency ratio	actual = optimal (=1) vs actual $\gg$ optimal ($\to 0$)	Ex 5
H	Tool precision / recall	over-calling (low precision) vs under-calling (low recall)	Ex 6
I	Safety lexicographic score	one violation wipes a "successful" run	Ex 7
J	Inter-rater agreement $\alpha$	agreement vs chance-level ($\alpha \to 0$)	Ex 8
K	Real-world word problem	end-to-end flight-booking pipeline	Ex 9
L	Exam twist	comparing two agents with overlapping error bars	Ex 10

Each example below is tagged with the cell(s) it fills.

---

---

Example 1 — Compounding failure (cells A, B)

Forecast: guess before reading — is the whole pipeline closer to 95%, 60%, or lower than both?

Why a figure here? Because the whole point is geometric: each surviving fraction multiplies the bar down. Look at the amber bar in the figure shrinking at every stage — that shrinking is the product.

1. Write the model. End-to-end success needs every step to succeed, and steps are independent, so the per-step probabilities $P_i$ multiply: $P(\text{success})={\prod }_{i=1}^6{P}_i.$ Why this step? We use a product, not a sum, because "and" over independent events means multiply. A sum would let one strong step rescue a weak one, which is false — a broken parser stops everything.

2. Plug in the good case. $P=0.95^5\times 0.60=0.77378\dots \times 0.60=\mathrm{0.46427.}$ Why this step? Five identical factors collapse to $0.95^5$; then one more factor for the flaky parser.

3. Now the degenerate case (cell B): set the parser to $0$: $P=0.95^5\times 0=0.$ Why this step? A single zero factor annihilates the whole product — this is the mathematical shadow of "if step 3 fails, step 5 is unreachable" from the parent note. There is no partial credit in a strict pipeline.

Verify: Sanity — the answer $0.4643$ is below both $0.95$ and $0.60$. That must be true: multiplying numbers below 1 always shrinks. And $0$ is unarguable. ✓

---

Example 2 — Success rate with confidence interval, small vs large $n$ (cell C)

Forecast: the point estimate is the same 60% both times — so what actually changes?

1. Point estimate. $\hat{p}=k/n=60/100=0.60$. Why this step? $\hat{p}$ is just the fraction that worked; it's our best single guess for the true rate.

2. Standard error — how jittery that guess is: $SE=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}=\sqrt{\frac{0.6\times 0.4}{100}}=\sqrt{0.0024}=\mathrm{0.0490.}$ Why this step? We use the binomial SE because each episode is an independent yes/no trial with the same probability — exactly the coin-flip setup the formula was built for.

3. 95% interval. Multiply SE by $1.96$ (the width that captures the middle 95% of a bell curve) and put it either side of $\hat{p}$: $CI=0.60\pm 1.96\times 0.0490=0.60\pm 0.096=[0.504,0.696].$ Why this step? We centre the interval on our best guess $\hat{p}$ and reach out $1.96$ standard errors each way because, by the central-limit theorem, $\hat{p}$ is approximately normally distributed around the true rate — and $\pm 1.96$ SE captures the central 95% of a normal curve, so the true rate lands inside this band 95% of the time.

4. Large $n$ (cell C, other end): $n=1000$, $\hat{p}=0.60$: $SE=\sqrt{\frac{0.24}{1000}}=0.01549,CI=0.60\pm 0.0304=[0.570,0.630].$ Why this step? $n$ sits under a square root in the denominator, so tenfold more episodes shrinks the bar by $\sqrt{10}\approx 3.16$, not by 10.

Verify: width dropped from $0.192$ to $0.0607$, a factor of $\sqrt{10}\approx 3.16$. ✓ This is why the parent said you need $n\ge 100$ to tell 60% from 70% apart. Units: all dimensionless probabilities. ✓

---

Example 3 — The boundary cases $\hat{p}=1$ and $\hat{p}=0$ (cell D)

Forecast: the SE formula will give the same surprising number in both cases. Guess what.

1. Case (a) estimate. $\hat{p}=20/20=1.0$.

2. Plug into SE: $SE=\sqrt{\frac{1.0\times (1-1.0)}{20}}=\sqrt{0}=0.$ Why this step? With $\hat{p}=1$, the factor $(1-\hat{p})=0$, so the formula reports zero uncertainty.

3. Naive CI: $1.0\pm 0=[1.0,1.0]$. Why this is wrong: the formula claims we are certain the true rate is exactly 100%. But 20 successes could easily come from a true rate of, say, 90%. The normal approximation collapses at the boundary because the bell-curve assumption fails when there is no observed variance.

4. The fix — Rule of Three (a quick, honest bound). When you see 0 failures in $n$ trials, the true failure rate could be as high as roughly $3/n$: $\text{plausible failure}\lesssim \frac{3}{20}=0.15\Rightarrow \text{true success}\gtrsim 0.85,$ so the honest interval is about $[0.85,1.0]$ instead of the fake $[1,1]$. Why this step? The bound comes from asking "what is the highest true failure rate $q$ that still makes seeing zero failures believable?" The chance of zero failures in $n$ independent trials is $(1-q{)}^n$; we call an outcome "believable" if that chance is at least 5%. Solving $(1-q{)}^n=0.05$ gives $q=1-0.05^{1/n}$, and since $\ln (0.05)\approx -3$, this is very close to $q\approx 3/n$ for reasonable $n$. So $3/n$ is the honest ceiling on the failure rate that a zero-failure run can rule out — no more, no less.

5. Case (b), the mirror $\hat{p}=0$: now $0/20=0$, and $SE=\sqrt{\frac{0\times (1-0)}{20}}=\sqrt{0}=0,$ giving the equally-fake $[0,0]$. Applying the Rule of Three to the successes this time: true success could be as high as $3/20=0.15$, so the honest interval is about $[0.0,0.15]$. Why this step? Zero successes is not proof the agent never works — it just hasn't shown a win in 20 tries; the honest ceiling is $0.15$.

Verify: $3/20=0.15$; the perfect run's honest floor $1-0.15=0.85$ and the zero run's honest ceiling $0.15$ both sit a full $0.15$ away from the fake boundary — exactly flagging that extreme small-sample scores are not proof. ✓

---

Example 4 — Discounted reward across all $\gamma$ limits (cells E, F)

Forecast: at $\gamma =0$ does the agent care about anything beyond the first step?

Recall the definition: $R={\sum }_{t=1}^T{\gamma }^{t-1}{r}_t$ — sum the rewards from step $1$ up to the horizon $T$, but each later one is faded by an extra factor of $\gamma$.

Why a figure? The cyan curve shows ${\gamma }^{t-1}$ collapsing toward zero for small $\gamma$ and staying flat at $1$ for $\gamma =1$ — you can see which rewards survive.

1. Case (a), $\gamma =0.9$, $T=4$: $R=10+0.9\cdot 5+0.9^2\cdot 2+0.9^3\cdot 1=10+4.5+1.62+0.729=\mathrm{16.849.}$ Why this step? The exponent is $t-1$, so the first reward ($t=1$) is undiscounted (${\gamma }^0=1$).

2. Case (b), the degenerate $\gamma =0$: every ${\gamma }^{t-1}$ is $0$ except ${\gamma }^0=1$ (we take $0^0=1$ by convention here): $R=10+0+0+0=10.$ Why this step? $\gamma =0$ means "only now matters" — a myopic agent that ignores the future entirely.

3. Case (c), the limit $\gamma =1$: no discounting, plain sum over the horizon $T=4$: $R=10+5+2+1=18.$ Why this step? $\gamma =1$ treats a reward in step 1000 as valuable as one now — fine for short finite tasks, dangerous for infinite ones.

4. Infinite horizon $T=\infty$, constant reward $r=1$, $\gamma =0.9$ (cell F): the geometric series $R={\sum }_{t=1}^{\infty }0.9^{t-1}\cdot 1=\frac{1}{1-0.9}=10.$ Why this step? Discounting guarantees a finite sum even over infinitely many steps ($T\to \infty$) — the parent's "mathematical convenience" reason, made concrete. With $\gamma =1$ this same sum would blow up to infinity.

Verify: $16.849$ (a) sits between $10$ (b) and $18$ (c), as it must for $\gamma$ between $0$ and $1$. The infinite series $1/(1-0.9)=10$ matches by the geometric-series formula. ✓

---

Example 5 — Efficiency ratio at both extremes (cell G)

Forecast: which way does the ratio move as an agent dawdles — up or down?

1. Agent A: $\frac{8}{8}=1.0$. Why this step? Optimal-over-actual equals 1 exactly when the agent is optimal — the upper limit, a clean ceiling.

2. Agent B: $\frac{8}{40}=0.20$. Why this step? Five times as many steps means one-fifth the efficiency; the ratio can never exceed 1 and slides toward 0 as wandering grows.

3. Limiting behaviour: as ${\text{steps}}_{\text{actual}}\to \infty$, efficiency $\to 0$. It is bounded in $(0,1]$.

Verify: $8/8=1$, $8/40=0.2$, both in $(0,1]$. ✓ A perfect agent gives 1; an infinitely wasteful one approaches 0. ✓

---

Example 6 — Tool precision vs recall, over- and under-calling (cell H)

Forecast: which agent looks better on precision, and which on recall? They disagree!

1. Agent X (over-caller): $\text{Precision}=\frac{3}{5}=0.60,\text{Recall}=\frac{3}{3}=\mathrm{1.0.}$ Why this step? X hit all the needed calls (recall 1) but wasted 2 extra ones (precision drops to 0.6).

2. Agent Y (under-caller): $\text{Precision}=\frac{2}{2}=1.0,\text{Recall}=\frac{2}{3}=\mathrm{0.667.}$ Why this step? Y never wasted a call (precision 1) but missed one needed call (recall 0.667).

3. Interpretation: precision and recall trade off. A single number that balances them is the F1 score, the harmonic mean: $F1=\frac{2\cdot P\cdot R}{P+R}.$ For X: $F1=\frac{2(0.6)(1.0)}{1.6}=0.75$. For Y: $F1=\frac{2(1.0)(0.667)}{1.667}=0.80$.

Verify: each ratio lies in $[0,1]$; X wastes so its precision $<1$; Y under-covers so its recall $<1$. Y's higher F1 (0.80 > 0.75) reflects that missing one call hurts less here than wasting two. ✓

---

Example 7 — Lexicographic safety wipes a "win" (cell I)

Forecast: should a single deletion cost 2% or 100% of the score?

1. Naive safety score: $1-\frac{1}{50}=\mathrm{0.98.}$ Why this step? It treats every action equally — but a deletion is not a rounding error, it is a red line.

2. Lexicographic score (safety first, then success): $\text{Score}=\{\begin{array}{ll}0& \text{if any hard constraint violated}\\ \text{task success}& \text{otherwise.}\end{array}$ Since a hard constraint was violated, $\text{Score}=0$ regardless of the task success. Why this step? "Delete files" was a hard constraint. Ordering safety above success means one breach zeroes the run — an agent that fails safely beats one that succeeds destructively.

Verify: naive $0.98$ vs lexicographic $0$ — the gap is the whole lesson of the parent's [!mistake] callout. The deletion flips the run to 0. ✓

---

Example 8 — Inter-rater agreement at chance (cell J)

Forecast: if raters agree only as often as random guessing would, what should $\alpha$ be?

Recall $\alpha =1-\frac{D_o}{D_e}$: it compares how much raters actually disagree against how much they'd disagree by pure chance.

1. Compute: $\alpha =1-\frac{0.10}{0.50}=1-0.20=\mathrm{0.80.}$ Why this step? $D_o$ well below $D_e$ means raters agree far more than chance, so $\alpha$ climbs toward 1. The parent's threshold $\alpha \ge 0.8$ is exactly met here.

2. Degenerate case: if $D_o=D_e$ (raters no better than chance), $\alpha =1-1=0$. Why this step? $\alpha =0$ is the "coin-flip" floor — the rubric carries no shared meaning; fix the rubric before trusting any human scores.

Verify: $1-0.1/0.5=0.80$; and $D_o=D_e\Rightarrow \alpha =0$. Both sit in the expected range: $0.80$ meets the reliability bar, $0$ is chance-level. Interpretation: at $\alpha =0.80$ we may trust the averaged human scores; at $\alpha =0$ we must rewrite the rubric before any evaluation number means anything. ✓

---

Example 9 — Real-world word problem: the flight-booking pipeline (cell K)

Forecast: will the predicted product land inside the measured error bars?

1. Predicted success (product model). Multiply the six $P_i$: $P=0.99\times 0.97\times 0.85\times 0.95\times 0.98\times 0.90=\mathrm{0.68388.}$ Why this step? Same "and over independent steps ⇒ multiply" logic as Example 1; the 0.85 parser is the weakest link.

2. Measured estimate: $\hat{p}=118/200=0.59$.

3. Measured CI: $SE=\sqrt{\frac{0.59\times 0.41}{200}}=\sqrt{0.00120950}=0.03478,$ $CI=0.59\pm 1.96\times 0.03478=0.59\pm 0.0682=[0.522,0.658].$ Why this step? We build the honest error bar so we can ask whether the theoretical prediction is plausible.

4. Compare: predicted $0.684$ lies just above the upper bound $0.658$. Why this step? The prediction falls outside the CI, signalling the steps are not truly independent — real pipelines have correlated failures (a bad parse cascades). The product model is an optimistic ceiling.

Verify: product $=0.68388$; $\hat{p}=0.59$; CI upper $=0.658<0.684$, so the model overestimates — a real, interpretable finding. ✓

---

Example 10 — Exam twist: are two agents really different? (cell L)

Forecast: 65% vs 72% looks decisive — but with only 100 episodes each, is it?

1. Agent P: $\hat{p}=0.65$, $SE=\sqrt{\frac{0.65\times 0.35}{100}}=0.0477,CI=0.65\pm 0.0935=[0.557,0.743].$

2. Agent Q: $\hat{p}=0.72$, $SE=\sqrt{\frac{0.72\times 0.28}{100}}=0.0449,CI=0.72\pm 0.0880=[0.632,0.808].$

3. Overlap check: P's interval reaches up to $0.743$; Q's reaches down to $0.632$. Since $0.632<0.743$, the intervals overlap. Why this step? Overlapping 95% intervals mean we cannot confidently declare Q better at $n=100$ — the apparent 7-point gap is within noise.

4. The fix: increase $n$. To halve the bar widths and separate them, quadruple the episodes (since $SE\propto 1/\sqrt{n}$).

Verify: overlap region is $[0.632,0.743]$, non-empty ⇒ not separable; and $SE\propto 1/\sqrt{n}$ means $4\times$ episodes ⇒ $2\times$ tighter bars. ✓ This is the parent's warning ("you need $n\ge 100$ to distinguish 60% vs 70%") made quantitative.

---

Recall Self-check

Why does a single 0% step zero the whole pipeline product? ::: Because independent "and" events multiply, and any zero factor annihilates a product — mathematically mirroring an unreachable downstream step. Why is a perfect small-sample score $\hat{p}=1$ misleading? ::: The binomial SE becomes $\sqrt{0}=0$, falsely claiming certainty; use the Rule of Three ($\approx 3/n$) for an honest bound. What does $\gamma =0$ mean for an agent? ::: Total myopia — only the immediate reward $r_1$ counts; all future terms vanish. When can't you declare Agent Q better than Agent P? ::: When their 95% confidence intervals overlap; the gap is within sampling noise. Why does lexicographic scoring send a task-successful-but-unsafe run to 0? ::: Safety is ordered above success, so any hard-constraint breach overrides the task reward entirely.

Prerequisite threads if any step felt fast: Benchmark Design, Human Evaluation Protocols, A/B Testing, Tool Use in Language Models, Reinforcement Learning Basics, Safety Alignment.