Autonomous agent evaluation

Think of it like evaluating a chess player: you can't judge them on one move in isolation—you need to see the full game, their strategy, how they adapt to opponent moves, and whether they win.

What We're Actually Measuring

When we evaluate autonomous agents, we care about:

==Task success rate – Did the agent achieve the goal? (binary or partial credit) Efficiency – How many steps/tokens/API calls did it take? Generalization – Does it work on unseen tasks/environments? Safety – Did it avoid harmful actions, respect constraints? Robustness== – Does it recover from errors, ambiguous instructions, noisy observations?

Why Evaluation is Hard: The Multi-Step Problem

The agent must:

1. Navigate to a travel site
2. Fill search form (origin, destination, date)
3. Parse search results
4. Filter by price
5. Select a flight
6. Complete booking

Challenge: If step 3 fails (parsing error), the agent can't reach step 5. Success at the end depends on every previous step. Traditional metrics (accuracy per action) don't capture this compounding failure problem.

Why This Matters

If each step has90% success rate and you need 10 steps:

Even high per-step accuracy → low task success. This is why we need end-to-end evaluation on complete task trajectories.

Key Evaluation Paradigms

1. Benchmark Environments

==Simulated environments== where we can reset, control, and measure precisely.

Why this step?

• We need a reproducible environment (can't use real websites—they change)
• We can instrument everything: exact actions taken, hidden state, ground truth goals
• We can run hundreds of trials cheaply

Metrics:

• Task success rate (binary: goal achieved or not)
• Step efficiency: $\text{efficiency} = \frac{\text{steps}_{\text{optimal}}}{\text{steps}_{\text{actual}}}$
• Token usage (for LLM-based agents)

Standard error (assuming binomial):

95% confidence interval:

Why? Small $n$ → large error bars. You need $n \geq 100$ to distinguish 60% vs 70% success reliably.

Why this step? We derive from the binomial distribution because each trial is independent with fixed success probability $p$. The CLT gives us the normal approximation for large $n$.

2. Human-in-the-Loop Evaluation

For tasks where success is subjective or hard to formalize.

A human evaluator judges:

• Was the response empathetic? (1-5 scale)
• Did it resolve the issue? (yes/no)
• Was it policy-compliant? (yes/no)

Aggregation: Average across multiple evaluators, compute inter-rater reliability (Krippendorff's α).

where $D_o$ = observed disagreement, $D_e$ = expected disagrement by chance.

Why this step? If evaluators don't agree ($\alpha < 0.8$), the task definition is ambiguous—fix the rubric first.

3. Tool-Use Evaluation

Measure appropriateness and correctness of tool calls.

Good trajectory:

Thought: I'll write a Python function to find primes
Action: execute_python("def nth_prime(n): ...")
Observation: 541
Answer: The 100th prime is 541

Metrics:

• Tool call accuracy: Did it call the right tool with valid arguments?
• Result correctness: Is the final answer right?
• Efficiency: Did it use unnecessary tools? (e.g., calling a calculator for 2+2)

Formula for Tool Precision/Recall:

Why this step? Precision measures waste (hallucinated tools, wrong args). Recall measures if the agent found the right tools for the job.

Common Evaluation Benchmarks

| Benchmark | Domain | Task Type | Metric | |-----------|-----------|-----| | ==WebArena | Web Navigation | Multi-step web tasks | Task success rate | | SWE-bench | Software Eng | Resolve GitHub issues | % fixed | | HotpotQA | QA with tools | Multi-hop reasoning + Wikipedia | Exact match F1 | | GAIA | General assistance | Real-world assistant tasks | Human eval score | | ToolBench== | Tool use | API calling for tasks | Tool call accuracy + task success |

Agent task:

1. Read issue description
2. Navigate codebase
3. Locate bug
4. Write patch
5. Verify with tests

Evaluation:

• Does the patch pass the issue's test?
• Does it pass all other existing tests (no regressions)?

Success rate: Top agents ~15-20% (humans ~80%)—shows how hard real-world SE is.

Why this step? Using real repos ensures ecological validity: if an agent succeds here, it might actually be useful. Synthetic benchmarks can be gamed.

Deriving an Evaluation Metric: Expected Discounted Reward

For agents in environments with partial credit (not just binary success).

where $\gamma \in [0,1]$ is the discount factor.

Why discount?

• Time preference: Earlier rewards are more valuable (finish sooner)
• Uncertainty: Future is less certain—heavily discount distant rewards
• Mathematical convenience: Ensures finite sum even for infinite horizons

Example calculation: Rewards: [10, 5, 2, 1] $\gamma = 0.9$

Why this step? We sum because rewards are additive. We discount geometrically because each step adds one layer of uncertainty ($\gamma^t$ compounds the discount).

Evaluation: Run agent on N episodes, compute:

with standard error:

Safety Evaluation: Measuring What Agents Shouldn't Do

Safety constraints:

• Don't delete files
• Don't leave desktop
• Don't access hidden system folders

Evaluation:

• Hard constraint violations: Count actions that break rules
• Safety score: $1 - \frac{\text{violations}}{\text{total actions}}$

Why it feels right: Task success is the primary metric, easy to focus on.

The fix: Separately track safety violations. An agent that deletes your files while organizing them is worse than one that fails the task safely. Use a lexicographic ordering: safety first, then success.

---

## Evaluation Design Principles

### 1. Compositionality: Test Skills in Isolation + Combination

Why? If an agent fails level 3, you can diagnose: is it the tool-calling (level 1), reasoning (level 2), or parsing (level 3)?

### 2. Distribution Shift: Test Generalization

Train/test split for agents:

• In-distribution: Same task types, domains

• Out-of-distribution: New domains, task structures

• Adversarial: Edge cases, tricky constraints

Metric: Success rate drop from in-dist → OOD measures generalization gap.

---

## Ablation Studies: What Actually Matters?

To understand why an agent works, systematically remove components:

$$\Delta_{\text{component}} = \text{Score}_{\text{full}} - \text{Score}_{\text{without component}}$$ $$n \approx \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{\delta^2}$$

where $\bar{p} = \frac{p_A + p_B}{2}$ (pooled success rate).

Example: Detect 5% improvement from 50% baseline, $\alpha=0.05$, $\beta=0.2$:

Why this step? We need enough samples to distinguish signal from noise. Too few → can't tell if improvement is real or luck.

The problem: Some chores have many steps, like "make a sandwich" (get bread, get peanut butter, spread it, etc.). If the robot messes up any step, the whole sandwich is ruined. So you can't just check if it's good at spreading—you have to see if it can do the entire chore start to finish.

Different ways to test:

1. Fake kitchen: You build a pretend kitchen where you can easily see what the robot does and reset everything. This is like a video game level for the robot.
2. Report card: You watch the robot and give it grades on different skills (speed, neatness, safety).
3. Real kitchen: You let it try in your actual kitchen—scary but shows if it really works.

What you measure:

• Did it finish the chore? ✓
• How long did it take? ⏱️
• Did it break anything or make a mess? 🛡️
• Can it handle new chores it's never seen?🌟

The hard part: If your robot only works on the chores you trained it on, it's like a student who memorized answers without understanding—fails on new questions!

Connections

• Reinforcement Learning Basics – reward functions, episode structure
• LM Prompting Strategies – ReAct, Chain-of-Thought (what agents use internally)
• Tool Use in Language Models – how agents call functions/APIs
• Multi-Agent Systems – evaluating coordination, communication
• Benchmark Design – principles for creating good test suites
• Human Evaluation Protocols – inter-rater reliability, rubric design
• A/B Testing – online evaluation statistics
• Safety Alignment – constraint satisfaction, red-teaming

#flashcards/ai-ml

Why can't we evaluate agents with simple accuracy metrics? :: Agents take multi-step actions in dynamic environments. Success depends on entire trajectories, not individual decisions. One early mistake can cascade and prevent goal completion.

Formula for discounted cumulative reward :: $R = \sum_{t=1}^{T} \gamma^{t-1} r_t$ where γ ∈ [0,1] is discount factor, $r_t$ is reward at step t.

Concept Map

Hinglish (regional understanding)