6.2.9 · D5AI Agents & Tool Use
Question bank — Autonomous agent evaluation
This is a self-test deck for Autonomous Agent Evaluation. Each line below is a prompt ::: answer reveal. Read the prompt, commit to an answer out loud, then reveal. The answers are short arguments, not verdicts — if your reasoning differs from the justification, that gap is the thing to fix.
The figures below make three ideas visual — compounding success, exponential discounting, and shrinking confidence intervals. Refer back to them as you go.
True or false — justify
An agent with 95% accuracy on every individual action will succeed on a 20-step task about 95% of the time.
False. Success requires all steps to work, so . Per-step accuracy compounds multiplicatively into task success, so long trajectories collapse even with high step accuracy (see the blue curve in figure s01).
Raising per-step success from 90% to 95% cuts the 10-step failure rate by roughly a third.
True. Task success goes from to , so failure drops from ~65% to ~40% — a relative reduction of about 38%, not a halving. Still a large gain, because the exponent amplifies the improvement across all ten steps.
A 60% success rate measured over 10 episodes reliably means the agent is better than one measured at 55%.
False. With the standard error is about , so the two estimates overlap massively. You need roughly before a 5-point gap becomes distinguishable (figure s03 shows the interval shrinking with ).
An agent that completes the task while deleting one user file should still score higher than an agent that safely fails.
False under lexicographic scoring. Safety is the higher-priority key: any hard-constraint violation sends the score to , so the safe failure (score = 0 task success but no violation) is preferred over a "successful" but destructive run.
The discount factor only matters for infinite-horizon tasks.
False. Even in finite-horizon tasks encodes time preference — it rewards achieving the goal in fewer steps by weighting early rewards more heavily. It changes which trajectory scores best, not just convergence.
If all human evaluators give a task a 5/5, the metric is trustworthy regardless of rubric quality.
False. Perfect agreement can also mean the rubric is trivial, leading, or that raters copied a default. High inter-rater reliability is necessary but not sufficient; the rubric must actually discriminate good from bad behaviour.
Precision and recall of tool calls always move together.
False. They trade off. An agent that calls extra unneeded tools keeps recall (it still found the right ones) but lowers precision (waste); an agent that calls too few keeps precision high but drops recall (missed needed tools).
Testing an agent only on the full end-to-end task tells you everything you need to know.
False. End-to-end success hides where it fails. Compositional evaluation (skills in isolation, then combined) is needed to attribute failure to a specific capability rather than a mystery compounding collapse.
A benchmark that an agent scores 20% on is a bad benchmark because scores are so low.
False. Low scores on ecologically valid benchmarks like SWE-bench (top agents ~15–20%, humans ~80%) are informative — they reveal real difficulty and leave headroom. A benchmark saturated at 99% is the one that has stopped measuring anything.
Spot the error
"Our agent got 100% accuracy on the 'fill the search form' action, so web-booking success should be high."
The error is treating one sub-action's accuracy as the task metric. Booking is a chain (navigate → search → parse → filter → select → book); a perfect form-fill is useless if result parsing fails downstream. Measure end-to-end trajectory success, not one link.
"We computed a 70% success rate over 8 episodes and reported it as the agent's ability."
The error is reporting a point estimate with no uncertainty on a tiny sample. With the 95% confidence interval spans roughly ±30 points, so "70%" is nearly meaningless. Report the CI and raise .
"The agent finished the file-organization task, so we recorded safety score = task success = 1."
The error is conflating success with safety. Safety violations must be tracked separately by counting rule-breaking actions; a run can succeed at the goal while violating a hard constraint, and lexicographic scoring should then zero it.
"Recall = (correct tool calls) / (total tool calls)."
That formula is precision, not recall. Recall divides correct calls by the number of tool calls needed for the optimal solution — it measures coverage of the right tools, not the purity of the calls made.
"We used a live e-commerce website so the benchmark reflects real usage."
The error is losing reproducibility. Real sites change between runs, so trials aren't comparable and you can't reset to ground truth. Benchmarks like WebArena use a controlled simulated site precisely to make results repeatable and instrumentable.
"Krippendorff's , so we averaged the human scores and moved on."
The error is trusting an unreliable aggregate. The threshold is the conventional cutoff for "reliable enough to draw conclusions" (values 0.667–0.8 are only tentatively usable), so signals evaluators barely agree beyond chance. Averaging noise doesn't fix it — you must sharpen the rubric before the numbers mean anything.
"Discounted reward with and rewards [10,5,2,1] equals ."
The error is forgetting the discount weights. It is ; each later reward is shrunk by a growing power of , so the sum is below the undiscounted total (figure s02 shows the shrinking weights).
Why questions
Why do we multiply per-step probabilities instead of averaging them for task success?
Because the task only succeeds if every step succeeds, and (assuming independence) the joint probability of a conjunction of events is their product. Averaging would describe a typical single step, not the survival of the whole chain.
Why derive success-rate confidence intervals from the binomial distribution?
Because each episode is an independent trial with a fixed success probability — exactly a binomial setup. The Central Limit Theorem then justifies the normal approximation (where is the standard error, the standard deviation of the estimate ) for large .
Why is safety scored lexicographically (safety first, then success) rather than as a weighted sum?
A weighted sum lets high task success "buy back" a constraint violation, which is exactly what we forbid — deleting files while organizing them should never be redeemable. Lexicographic ordering makes any hard violation dominate regardless of task reward.
Why discount future rewards geometrically (each step multiplied by ) rather than by a fixed subtraction?
Two clean reasons. (1) Time preference: multiplying by per step makes a reward that arrives later worth strictly less, so the metric prefers agents that finish sooner. (2) Convergence: a geometric weighting sums to a finite value even over an infinite horizon, whereas subtracting a fixed amount would eventually push rewards negative and never converge.
Why prefer real GitHub issues (SWE-bench) over synthetic bug benchmarks?
For ecological validity — success on genuine repository issues predicts real usefulness, whereas synthetic tasks can be pattern-matched or gamed. Real code brings the messy context (existing tests, regressions) that a synthetic task strips away.
Why measure tool-call precision and recall instead of one combined accuracy?
They capture different failures: precision penalizes waste and hallucinated/wrong tool calls, recall penalizes missing tools the task actually required. A single number would let a wasteful-but-thorough agent look identical to a lean-but-incomplete one.
Why do we need human-in-the-loop evaluation at all if we have automated metrics?
For tasks where success is subjective or not formalizable (empathy, policy-compliance in customer service), no automatic checker exists. Humans supply the ground truth, and inter-rater reliability tells us whether that ground truth is consistent.
Edge cases
What does the success rate CI look like when (zero successes in trials)?
The naive collapses to a zero-width interval, which is wrong — you clearly can't be certain the true rate is exactly 0. This is a known failure of the normal approximation at the boundary; use an exact/Wilson interval instead.
What is the discounted reward when ?
Only the first step counts: with the convention , the term survives and every later term is multiplied by . The agent is scored purely on immediate reward — a maximally myopic evaluation with no credit for future payoff.
What is the discounted reward when ?
It becomes the plain undiscounted sum , giving equal weight to all steps. This risks non-convergence over infinite horizons and removes any incentive to finish sooner.
What happens to the safety score when the agent takes zero actions?
The denominator is , so the ratio is undefined (division by zero). A no-op agent has no meaningful safety score under this formula — you must handle the zero-action case explicitly rather than let it slip through.
How should efficiency behave if the agent finishes in fewer steps than the assumed optimum?
The ratio would exceed 1, signalling the "optimal" reference was actually beatable — either the baseline is wrong or a genuinely shorter path exists. It's a red flag to re-derive the optimal-step count, not a bonus to celebrate.
What does a task-success metric report for an agent that never terminates (loops forever)?
Under binary success it scores 0 (goal never reached), but only if the harness enforces a step/time budget to cut the loop off. Without a timeout the evaluation itself hangs, so a max-horizon cap is a required part of the metric definition.
If every step has 100% success probability, what is the compounded task success?
Exactly , since for any number of steps. This is the only case where the compounding problem disappears — any per-step probability below 1 decays as the trajectory lengthens.
Recall One-line takeaways
Compounding ::: Task success is the product of per-step successes, so long chains collapse even at high step accuracy. Sample size ::: Small gives huge confidence intervals; you need ~100+ episodes to trust a few-point gap. Safety ordering ::: Any hard-constraint violation zeroes the score, lexicographically above task success. Discounting ::: With (step index starting at 0), the weight shrinks later rewards — encoding time preference and keeping infinite sums finite.