Autonomous agent evaluation

6.2.9 · AI-ML › AI Agents & Tool Use

Socho jaise kisi chess player ko judge karna: ek akele move par nahi kar sakte—poora game dekhna padega, unki strategy, opponent moves ke saath adaptation, aur yeh ki jeet hai ya nahi.

Hum Actually Kya Measure Kar Rahe Hain

Jab hum autonomous agents ko evaluate karte hain, hume yeh cheezein important lagti hain:

==Task success rate – Kya agent ne goal achieve kiya? (binary ya partial credit) Efficiency – Kitne steps/tokens/API calls lage? Generalization – Kya yeh unseen tasks/environments par kaam karta hai? Safety – Kya usne harmful actions se bachte hue constraints respect kiye? Robustness== – Kya yeh errors, ambiguous instructions, noisy observations se recover karta hai?

Evaluation Hard Kyun Hai: The Multi-Step Problem

Agent ko yeh karna hai:

1. Travel site par navigate karo
2. Search form fill karo (origin, destination, date)
3. Search results parse karo
4. Price ke hisaab se filter karo
5. Ek flight select karo
6. Booking complete karo

Challenge: Agar step 3 fail ho jaye (parsing error), toh agent step 5 tak pahunch nahi sakta. End mein success har ek pichle step par depend karti hai. Traditional metrics (accuracy per action) is compounding failure problem ko capture nahi kar paate.

Yeh Kyun Matter Karta Hai

Agar har step ki 90% success rate hai aur 10 steps chahiye:

High per-step accuracy bhi → low task success. Isliye hume complete task trajectories par end-to-end evaluation chahiye.

Key Evaluation Paradigms

1. Benchmark Environments

==Simulated environments== jahan hum precisely reset, control, aur measure kar sakte hain.

Yeh step kyun?

• Hume ek reproducible environment chahiye (real websites use nahi kar sakte—woh change hoti rehti hain)
• Hum sab kuch instrument kar sakte hain: exact actions liye, hidden state, ground truth goals
• Hum saikdon trials saste mein run kar sakte hain

Metrics:

• Task success rate (binary: goal achieve hua ya nahi)
• Step efficiency: $\text{efficiency}=\frac{{\text{steps}}_{\text{optimal}}}{{\text{steps}}_{\text{actual}}}$
• Token usage (LLM-based agents ke liye)

Standard error (binomial assume karke):

95% confidence interval:

Kyun? Chota $n$ → bade error bars. 60% vs 70% success reliably distinguish karne ke liye $n\ge 100$ chahiye.

Yeh step kyun? Hum binomial distribution se derive karte hain kyunki har trial independent hai fixed success probability $p$ ke saath. CLT humein large $n$ ke liye normal approximation deta hai.

2. Human-in-the-Loop Evaluation

Un tasks ke liye jahan success subjective hai ya formalize karna mushkil hai.

Ek human evaluator judge karta hai:

• Kya response empathetic tha? (1-5 scale)
• Kya usne issue resolve kiya? (yes/no)
• Kya yeh policy-compliant tha? (yes/no)

Aggregation: Multiple evaluators mein average karo, inter-rater reliability compute karo (Krippendorff's α).

jahan $D_o$ = observed disagreement, $D_e$ = chance se expected disagreement.

Yeh step kyun? Agar evaluators agree nahi karte ($\alpha <0.8$), toh task definition ambiguous hai—pehle rubric theek karo.

3. Tool-Use Evaluation

Tool calls ki appropriateness aur correctness measure karo.

Achhi trajectory:

Thought: I'll write a Python function to find primes
Action: execute_python("def nth_prime(n): ...")
Observation: 541
Answer: The 100th prime is 541

Metrics:

• Tool call accuracy: Kya usne sahi tool valid arguments ke saath call kiya?
• Result correctness: Kya final answer sahi hai?
• Efficiency: Kya usne unnecessary tools use kiye? (jaise 2+2 ke liye calculator call karna)

Tool Precision/Recall ka Formula:

Yeh step kyun? Precision waste measure karta hai (hallucinated tools, wrong args). Recall measure karta hai ki agent ne kaam ke liye sahi tools dhunde ya nahi.

Common Evaluation Benchmarks

| Benchmark | Domain | Task Type | Metric | |-----------|-----------|-----| | ==WebArena | Web Navigation | Multi-step web tasks | Task success rate | | SWE-bench | Software Eng | GitHub issues resolve karo | % fixed | | HotpotQA | QA with tools | Multi-hop reasoning + Wikipedia | Exact match F1 | | GAIA | General assistance | Real-world assistant tasks | Human eval score | | ToolBench== | Tool use | Tasks ke liye API calling | Tool call accuracy + task success |

Agent task:

1. Issue description padho
2. Codebase navigate karo
3. Bug dhundo
4. Patch likho
5. Tests se verify karo

Evaluation:

• Kya patch issue ka test pass karta hai?
• Kya yeh baaki saare existing tests pass karta hai (koi regressions nahi)?

Success rate: Top agents ~15-20% (humans ~80%)—dikhata hai real-world SE kitna hard hai.

Yeh step kyun? Real repos use karne se ecological validity ensure hoti hai: agar koi agent yahan succeed karta hai, toh shayad actually useful bhi hoga. Synthetic benchmarks ko game kiya ja sakta hai.

Ek Evaluation Metric Derive Karna: Expected Discounted Reward

Un agents ke liye jo partial credit wale environments mein hain (sirf binary success nahi).

jahan $\gamma \in [0,1]$ discount factor hai.

Discount kyun?

• Time preference: Pehle ke rewards zyada valuable hain (jaldi finish karo)
• Uncertainty: Future kam certain hai—door ke rewards ko zyada discount karo
• Mathematical convenience: Infinite horizons ke liye bhi finite sum ensure karta hai

Example calculation: Rewards: [10, 5, 2, 1] $\gamma =0.9$

Yeh step kyun? Hum sum karte hain kyunki rewards additive hain. Hum geometrically discount karte hain kyunki har step uncertainty ki ek layer add karta hai (${\gamma }^t$ discount ko compound karta hai).

Evaluation: Agent ko N episodes par run karo, compute karo:

standard error ke saath:

Safety Evaluation: Yeh Measure Karna Ki Agent Kya Nahi Kare

Safety constraints:

• Files delete mat karo
• Desktop se bahar mat jao
• Hidden system folders access mat karo

Evaluation:

• Hard constraint violations: Rules todne wale actions count karo
• Safety score: $1-\frac{\text{violations}}{\text{total actions}}$

Kyun sahi lagta hai: Task success primary metric hai, isi par focus karna easy hai.

Fix: Safety violations alag se track karo. Jo agent organize karte waqt tumhari files delete kare, woh us agent se worse hai jo task safely fail kare. Lexicographic ordering use karo: pehle safety, phir success.

---

## Evaluation Design Principles

### 1. Compositionality: Skills Ko Isolation + Combination Mein Test Karo

Kyun? Agar agent level 3 fail kare, toh diagnose kar sakte ho: kya problem tool-calling mein hai (level 1), reasoning mein (level 2), ya parsing mein (level 3)?

### 2. Distribution Shift: Generalization Test Karo

Agents ke liye train/test split:

• In-distribution: Same task types, domains

• Out-of-distribution: Naye domains, task structures

• Adversarial: Edge cases, tricky constraints

Metric: In-dist → OOD success rate drop generalization gap measure karta hai.

---

## Ablation Studies: Actually Kya Matter Karta Hai?

Yeh samajhne ke liye ki agent kyun kaam karta hai, systematically components remove karo:

$${\Delta }_{\text{component}}={\text{Score}}_{\text{full}}-{\text{Score}}_{\text{without component}}$$ $$n\approx \frac{2(z_{1-\alpha /2}+z_{1-\beta }{)}^2{\sigma }^2}{{\delta }^2}$$

jahan $\bar{p}=\frac{p_A+p_B}{2}$ (pooled success rate) hai.

Example: 50% baseline se 5% improvement detect karna, $\alpha =0.05$, $\beta =0.2$:

Yeh step kyun? Signal aur noise mein distinguish karne ke liye enough samples chahiye. Kam samples → nahi pata chalega ki improvement real hai ya luck.

Problem: Kuch kaamon mein bahut saare steps hote hain, jaise "sandwich banao" (bread lo, peanut butter lo, lagao, etc.). Agar robot koi bhi step galat kare, toh poori sandwich barbaad ho jaati hai. Toh tum sirf yeh nahi dekh sakte ki woh achi tarah spread karta hai—tumhe dekhna hai ki kya woh poora kaam start se finish tak kar sakta hai.

Test karne ke alag-alag tarike:

1. Fake kitchen: Tum ek naqli kitchen banate ho jahan tum easily dekh sako robot kya kar raha hai aur sab reset kar sako. Yeh robot ke liye ek video game level jaisa hai.
2. Report card: Tum robot ko dekhte ho aur usse alag-alag skills par grades dete ho (speed, neatness, safety).
3. Real kitchen: Usse apni asli kitchen mein try karne dete ho—daraaona hai par dikhata hai ki kya woh sach mein kaam karta hai.

Kya measure karte ho:

• Kya usne kaam finish kiya? ✓
• Kitna time laga? ⏱️
• Kya usne kuch toda ya gandagi ki? 🛡️
• Kya woh naye kaam handle kar sakta hai jो usne kabhi nahi dekhe? 🌟

Mushkil hissa: Agar tumhara robot sirf usi kaam par kaam karta hai jis par tumne train kiya, toh yeh us student jaisa hai jisne bina samjhe answers yaad kar liye—naye questions par fail ho jaata hai!

Connections

• Reinforcement Learning Basics – reward functions, episode structure
• LM Prompting Strategies – ReAct, Chain-of-Thought (jo agents internally use karte hain)
• Tool Use in Language Models – agents functions/APIs kaise call karte hain
• Multi-Agent Systems – coordination, communication evaluate karna
• Benchmark Design – achhe test suites banane ke principles
• Human Evaluation Protocols – inter-rater reliability, rubric design
• A/B Testing – online evaluation statistics
• Safety Alignment – constraint satisfaction, red-teaming

#flashcards/ai-ml

Simple accuracy metrics se agents ko evaluate kyun nahi kar sakte? :: Agents dynamic environments mein multi-step actions lete hain. Success poori trajectories par depend karti hai, individual decisions par nahi. Ek early mistake cascade ho sakti hai aur goal completion rok sakti hai.

Discounted cumulative reward ka formula :: $R={\sum }_{t=1}^T{\gamma }^{t-1}{r}_t$ jahan γ ∈ [0,1] discount factor hai, $r_t$ step t par reward hai.

Concept Map