Exercises — Autonomous agent evaluation

(a) This is the task success rate — a binary "goal achieved / not achieved" measure, not per-step accuracy. (b) The point estimate is the fraction of successes: $\hat{p}=\frac{k}{n}=\frac{128}{200}=0.64$ Here $k$ = number of successful episodes, $n$ = total episodes. So $\hat{p}=64\%$.

(a) Human-in-the-loop — empathy is subjective and hard to formalize, so a human rates it (e.g. 1–5). See Human Evaluation Protocols. (b) Tool-use evaluation — we check the correctness and appropriateness of a tool call. See Tool Use in Language Models. (c) Benchmark environment — a reproducible, reset-able simulator we can instrument fully. See Benchmark Design.

Why multiply? "All must succeed" means the task succeeds only if step 1 succeeds and step 2 succeeds and … and step $n$ succeeds. For independent events, the probability of an "and" is the product of the probabilities — that is the definition of independence. So: $P(\text{success})={\prod }_{i=1}^{n}P(\text{step }i)=p^n$ (The symbol $\prod$ just means "multiply all these together", like $\sum$ means "add all these together".) For $n=4$: $0.95^4=0.81450625\approx 0.8145$ (about $81.5\%$). For $n=20$: $0.95^{20}\approx 0.3585$ (about $35.9\%$). What the figure shows and how to read it. The horizontal axis is the number of steps $n$; the vertical axis is the overall success $p^n$, running from $0$ at the bottom to $1$ at the top. Three curves fall from the top-left corner: magenta ($p=0.95$), violet ($p=0.90$), orange ($p=0.80$). The key visual is the shape: each curve leaves the top edge at $n=0$ (zero steps means nothing can fail, so success $=1$) and then droops — gently at first, then relentlessly — because every extra step multiplies in another factor smaller than $1$. Follow the magenta curve rightward with your eye: at $n=4$ it has only slipped to about $0.815$ (the first arrow), but by $n=20$ it has sagged all the way to $0.359$ (the second arrow), crossing below the $0.4$ gridline. The violet and orange curves sit underneath magenta everywhere and plunge even faster — the smaller the per-step rate, the steeper the droop. That downward bend, present in all three curves, is the compounding-failure phenomenon made visible: high per-step reliability still collapses into low whole-task reliability once enough steps chain together.

Where does the standard error come from? Each episode is a coin-flip: success (count $1$) with probability $p$, failure (count $0$) with probability $1-p$. A single such flip has variance $p(1-p)$ — you can check the extremes: if $p=0$ or $p=1$ the outcome never varies so variance is $0$, and it is largest at $p=0.5$ where the coin is most unpredictable. We estimate $\hat{p}$ by averaging $n$ independent flips, and averaging $n$ independent things divides their variance by $n$: $\text{Var}(\hat{p})=\frac{p(1-p)}{n}.$ The substitution we must make honest. This formula contains the true rate $p$, which we do not know — if we knew $p$ we wouldn't be estimating it. So we plug in our best guess $\hat{p}$ in its place, giving the approximation $\text{Var}(\hat{p})\approx \frac{\hat{p}(1-\hat{p})}{n}.$ This is legitimate because $\hat{p}\to p$ as $n$ grows, so for large $n$ the swapped-in value is close. The standard error is the square root of that variance — how much $\hat{p}$ would wobble across repeated 200-episode runs: $SE=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}=\sqrt{\frac{0.64\cdot 0.36}{200}}=\sqrt{0.001152}\approx 0.033941$ Why is a normal curve allowed here? The Central Limit Theorem says an average of many independent flips is approximately bell-shaped. The rule of thumb for the approximation to be trustworthy is $n\hat{p}>5$ and $n(1-\hat{p})>5$. Here $n\hat{p}=200(0.64)=128$ and $n(1-\hat{p})=72$ — both comfortably above $5$, so we're safe. The 95% interval uses the multiplier $1.96$ (covers the central $95\%$ of a normal curve): $CI=\hat{p}\pm 1.96\cdot SE=0.64\pm 1.96(0.033941)=0.64\pm 0.06653$ So $CI\approx [0.5735,0.7065]$, i.e. roughly $[57.3\%,70.7\%]$.

The discount factor $\gamma$ shrinks later rewards; step $t$ is weighted by ${\gamma }^{t-1}$: $R={\sum }_{t=1}^T{\gamma }^{t-1}{r}_t=8+0.8\cdot 4+0.8^2\cdot 2$ $R=8+3.2+1.28=12.48$ (See Reinforcement Learning Basics for where $\gamma$ comes from.)

Agent A (from 2.2): $C{I}_A\approx [0.5735,0.7065]$. Agent B: $S{E}_B=\sqrt{\frac{0.68\cdot 0.32}{200}}=\sqrt{0.001088}\approx 0.032985$. $C{I}_B=0.68\pm 1.96(0.032985)=0.68\pm 0.06465\approx [0.6154,0.7446]$ The intervals overlap heavily ($[0.5735,0.7065]$ vs $[0.6154,0.7446]$), so with $n=200$ we cannot conclude B is better — the 4-point gap is within noise. What the figure shows. It draws two horizontal bars, one per agent: the dot marks the point estimate and the whiskers reach out to $\pm 1.96SE$. Agent A's bar (magenta, lower) is centred at $0.64$; Agent B's bar (violet, upper) is centred at $0.68$. Look at where the two bars share horizontal ground — the shaded orange band between about $0.615$ and $0.707$: that band is the overlap, and its sheer width is the visual signal that the estimates are statistically tangled. When two error bars overlap that much, sliding one agent up or down within its own uncertainty could reverse the ranking, so we have no license to call B better. This is exactly the parent-note warning that small $n$ gives fat error bars. To separate 64% from 68% you'd need a much larger $n$ (or a paired A/B Testing design that cancels shared per-task variance).

• correct tool calls = 1 (execute_python)
• total tool calls = 2
• tool calls needed for optimal solution = 1 $\text{Precision}=\frac{\text{correct}}{\text{total}}=\frac{1}{2}=0.5$ $\text{Recall}=\frac{\text{correct}}{\text{needed}}=\frac{1}{1}=1.0$ Interpretation: Recall $=1.0$ → it found the right tool. Precision $=0.5$ → half its calls were wasted (search_web was unnecessary). Low precision flags waste / hallucinated tools; recall flags missing the right tool.

Lexicographic (safety-first) rule: $\text{Score}=\{\begin{array}{ll}0& \text{if any hard constraint violated}\\ \text{task success}& \text{otherwise}\end{array}$ One hard violation occurred → Score $=0$, regardless of task success. Naive safety-fraction: $1-\frac{\text{violations}}{\text{total actions}}=1-\frac{1}{12}\approx 0.9167$ Disagreement: the naive score says "$92\%$ safe — pretty good!", which rewards an agent that irreversibly deleted a user file. The lexicographic score refuses any credit once a hard constraint breaks — deleting files while "succeeding" is worse than failing safely. What the figure shows. It draws two stacked shelves: an upper magenta "SHELF 1: safety" and a lower violet "SHELF 2: success". The reading rule is drawn as arrows: you only drop your gaze to the success shelf after the safety shelf is confirmed clear. The orange arrow plunging from the safety shelf down to a "Score = 0" tag at the bottom is our case — one violation ejects the agent off the top shelf to a hard zero before success is ever consulted. The separate navy arrow (clean pass) is the only path that reaches the success shelf. See Safety Alignment.

The idea in words first. Break the agent into its three independent sub-skills, give each the kind of metric that matches its nature (a tool call is objective; a summary's quality is subjective), then run the whole pipeline end-to-end so a low final number can be traced back to whichever stage failed. Below is that mapping written as a narrative list (so it reads the same in any renderer, not just as a table):

• Sub-skill (i): search tool call. Metric = tool precision / recall on valid arguments (did it call the right tool with valid args?). Paradigm = tool-use evaluation, see Tool Use in Language Models.
• Sub-skill (ii): reading / retrieval. Metric = exact-match / F1 against known ground-truth passages. Paradigm = benchmark environment, see Benchmark Design.
• Sub-skill (iii): summary quality. Metric = human 1–5 quality score plus an inter-rater reliability check. Paradigm = human-in-the-loop, see Human Evaluation Protocols.

Why inter-rater reliability, and why Krippendorff's $\alpha$ specifically? A single human's 1–5 score is only trustworthy if other humans would score similarly — otherwise you're measuring the rater's mood, not the summary. So we need a reliability coefficient. We pick ==Krippendorff's $\alpha$== over Cohen's $\kappa$ because $\kappa$ handles exactly two raters and only nominal (categorical) labels, whereas summary grading here uses more than two raters on an ordinal 1–5 scale, and often has missing ratings (not every judge rates every item). Krippendorff's $\alpha$ is the one coefficient that gracefully covers all three of those (any number of raters, ordinal/interval data, missing values), which is precisely our situation.

Compose end-to-end: run full trajectories, apply the lexicographic guard (safety first), then report task success rate with a 95% CI. Report per-sub-skill scores alongside end-to-end, so a low end-to-end number can be traced to the failing stage — that's the whole point of compositionality: isolate to diagnose, combine to judge real usefulness.

The CI half-width is $1.96\cdot SE=1.96\sqrt{\frac{p(1-p)}{n}}$. We want this $\le 0.025$. Use worst-case $p(1-p)=0.25$: $1.96\sqrt{\frac{0.25}{n}}\le 0.025⟹\sqrt{\frac{0.25}{n}}\le \frac{0.025}{1.96}=0.012755$ Why is squaring both sides legal here? Squaring preserves an inequality only when both sides are non-negative — and they are: a square root is never negative, and $0.012755>0$. (If either side could be negative, squaring might flip the inequality, so this check matters.) Squaring: $\frac{0.25}{n}\le 0.00016270⟹n\ge \frac{0.25}{0.00016270}\approx 1536.6$ So $n\approx 1537$ episodes per agent (round up). Lesson: distinguishing small gaps costs thousands of trials — which is why practitioners prefer paired A/B Testing on shared tasks to cut variance dramatically.

A synthetic benchmark can be gamed: the agent may exploit templated patterns that don't exist in the wild, so high scores lack ecological validity. Real-task benchmarks (real GitHub issues verified by the issue's own tests plus no-regression on existing tests) measure whether the agent is actually useful. The $85\%$ vs $\sim 15$–$20\%$ gap is precisely the signal that our synthetic setup over-estimates capability. Ship decision should rest on real-task success with confidence intervals and a passing safety guard — not the friendly synthetic number.

Overall task success from per-step $p$ over $n$ steps ::: $p^n$ (steps independent, all must succeed) Standard error of a success proportion ::: $\sqrt{\hat{p}(1-\hat{p})/n}$ Discounted reward weight on step $t$ ::: ${\gamma }^{t-1}$ (first reward undiscounted) Why lexicographic scoring, not weighted blend ::: safety must never be bought back by task success Why real benchmarks over synthetic ::: ecological validity — synthetic benchmarks can be gamed