Foundations — Autonomous agent evaluation

Before you can read the parent note, you need to meet each piece of notation as a friend, not a stranger. We build them one at a time, each on top of the last. Nothing appears before it is defined.

1. The agent, the environment, and a step

Let's start with the simplest picture: a robot facing a world.

Look at the figure. The blue box is the agent. The navy cloud is the environment. The yellow arrow going out is an action; the green arrow coming back is an observation (what the agent now sees) plus a reward (a score for that step). This loop repeats.

Why the topic needs this: every metric on the parent page is really counting something inside this loop — how many loops (steps), whether the final loop reached the goal, whether any loop broke a rule.

Related idea in the vault: Tool Use in Language Models and Reinforcement Learning Basics both live inside exactly this loop.

2. Subscripts, the index $t$, and the reward symbol $r$

The parent writes things like $r_1,r_2,\dots ,r_T$ and ${\sum }_{i=1}^n$. Before those make sense, you need the idea of a subscript — and you need to know what the letter $r$ stands for.

Picture a row of numbered lockers. $r_1$ is the reward in locker 1, $r_2$ the reward in locker 2. The letter $t$ or $i$ is a placeholder for "some locker number" — when we write $r_t$ we mean "whichever step we're currently pointing at."

Why the topic needs this: an agent's journey is a list of steps. Subscripts let us name step 1, step 2, ... step $T$ without inventing a new letter each time.

3. The trajectory: a sequence of actions and rewards

The figure shows one trajectory as a chain of dots. Each dot is a step; the yellow number under it is the reward $r_t$ collected at that step. Success or failure is decided by the whole chain.

4. Probability $P(\cdot )$ and multiplying independent chances

The parent writes $P(\text{success})={\prod }_{i=1}^{n}P(\text{success at step }i)$. Two symbols hide here. (Here $n$ is just the number of steps — the same count we called $T$ in §3; the parent happens to write $n$ in this one formula.)

Why multiply, not add? Because to reach step 3 you must survive step 1 and step 2 and step 3. "And" of independent chances is a product. That is why chances compound: each new step is another factor below 1, dragging the total down.

The red curve shows overall success collapsing as the number of steps grows, even though each step is a healthy 90%. Read it as: long journeys are fragile.

5. The summation symbol $\sum$, an episode score, and the average

First a naming rule to keep counts straight:

Why the topic needs this: one trajectory's score $R_i$ is noisy luck. Running $N$ episodes and averaging gives the agent's typical performance. This is the backbone of A/B Testing and every benchmark table.

6. The hat $\hat{p}$: an estimate from data

Picture flipping a bent coin. The real bias $p$ is hidden inside the coin. After 100 flips you only have $\hat{p}$, your measured fraction $k/n$. The hat is a humble symbol: it admits "this is a guess."

7. Spread, standard error, and the $\pm$ confidence interval

The figure shows two agents' measured success rates as dots with error bars (the $\pm$ range). When the bars overlap, you cannot claim one agent beats the other — the difference might be luck. This is why the parent insists $n\ge 100$ episodes: more episodes shrink $SE$, shrinking the bars until they separate.

8. The discount factor $\gamma$ and the exponent ${\gamma }^{t-1}$

The parent writes $R={\sum }_{t=1}^T{\gamma }^{t-1}{r}_t$ (note this sums over steps $t$ within one episode, using lowercase $r_t$ from §2).

Why an exponent, and why this tool? Each extra step adds one more layer of uncertainty. Multiplying by $\gamma$ once per step means the doubt compounds — exactly the same "and-of-chances" logic as the product in §4, but now applied to value instead of survival. An exponent is the natural tool for "same shrink, repeated every step."

Deeper theory of $\gamma$ lives in Reinforcement Learning Basics.

9. Ratios: efficiency, precision, recall, safety score

Many parent metrics are just one count divided by another — a ratio answers "what fraction?"

How to read any ratio: the bottom is "how many chances there were," the top is "how many went the way we wanted." All four land between 0 and 1, so they're directly comparable. Precision/recall detail: Human Evaluation Protocols.

10. The piecewise brace and lexicographic ordering

The parent's safety score uses a big curly brace.

The prerequisite map

This feeds directly into the parent topic, and connects onward to Multi-Agent Systems and LM Prompting Strategies.

Equipment checklist