Foundations — Agent, environment, state, action, reward

This page builds every symbol the parent note uses, starting from a reader who has never seen an equation. We go one brick at a time. Nothing is used before it is drawn.

1. The loop itself (before any symbol)

Picture two boxes joined by two arrows.

Look at the figure. The left box is the agent (the decider). The right box is the environment (the world). One arrow goes agent → environment: the agent sends an action. Two arrows come back environment → agent: a new situation and a score.

Why we need this split. Learning only makes sense if there is a "me" that acts and a "not-me" that reacts. Without the boundary there is no notion of a choice having a consequence. This split is the seed of the whole topic.

2. Time steps and the little subscript $t$

The loop happens over and over. We label each turn with a time step.

Picture: a filmstrip. Each frame is one value of $t$. Frame $0$, frame $1$, frame $2$… The subscript just tells you which frame you are looking at.

Why the topic needs it. The agent's whole job is to connect what it does now to what happens later. "Later" only has meaning if we can number the turns. $t$ is that numbering.

3. State $s_t$ — a complete description of the situation

Picture: in the grid world below, the state is simply which cell the robot stands on. That single fact — the coordinates $(x,y)$ — is enough to describe the whole situation.

Why "complete" matters. If the snapshot leaves something out that affects the future, the agent is guessing blind. When the snapshot is complete we call the world fully observable. When the agent only sees part of it (poker, a robot with weak sensors), we are in partial observability.

4. Action $a_t$ — the agent's one lever

Picture: in the grid figure, the four arrows out of the robot — North, South, East, West — are the possible actions. The agent picks one arrow.

Actions come in flavours:

• Discrete — a finite menu: $\{\text{North},\text{South},\text{East},\text{West}\}$.
• Continuous — a dial that slides: heater power anywhere in $[0,1]$.

Why actions are "first-class". The agent cannot reach into the world and set the state or hand itself a reward. It can only choose $a_t$; everything else emerges as a consequence. This is why RL is called learning by trial and error.

5. Reward $r_t$ — the single score

Picture: in the grid figure, stepping onto the goal flag prints $+10$; every ordinary step prints $-1$; bumping a wall prints $-5$. One number per step.

Why a single number. To compare two behaviours the agent needs one score to rank them. A vector of scores gives no clear "better". Collapsing everything to one scalar is what makes "maximize" meaningful.

6. Reading the symbol $\sim$ — "is drawn from"

The parent note writes things like $a_t\sim \pi (a\mid s_t)$. We must earn that wiggle.

Picture: a spinner. $\sim$ says "spin it and take whatever it lands on." $\mid s$ says "we're using the spinner for situation $s$."

Why the topic needs randomness. Worlds are often unpredictable: the same action in the same state can lead to different next states. $\sim$ lets us say "the next state is sampled, not fixed."

7. Policy $\pi$ — the agent's strategy

Picture: a lookup card. On the left, the situation; on the right, the arrow to take (or a set of odds over arrows). The policy is that card.

Why $\pi$ and not just "brain". All of learning is really "find a better card." Naming the card $\pi$ lets us later say "find the best policy ${\pi }^{\ast }$" — see Policy and Value Functions.

8. Transition probability $P$ — the world's physics

Picture: from a cell in the grid, an action fans out into arrows to possible next cells, each labelled with a probability. In a clean grid one arrow has probability $1$; on ice, action "North" might land North $80\%$ and sideways $10\%$ each.

Why we write it as a probability. It gathers deterministic worlds (one arrow, probability $1$) and stochastic worlds (many arrows) under one notation. Whether the agent knows $P$ or not splits Model-Free vs Model-Based RL.

9. The Markov property — "the present is enough"

Picture: a chess board. To choose your next move you only need the current arrangement of pieces — not the order they arrived in. The present board is a sufficient summary of the past.

Why it is the whole reason states matter. If we designed the state well, we may throw away the entire history and still decide optimally. This assumption turns the loop into a Markov Decision Process (MDP).

10. Trajectory $\tau$ — the record of one run

Chain the symbols in loop order and you get the story of one episode:

$\tau :s_0,a_0,r_0,s_1,a_1,r_1,s_2,a_2,r_2,\dots$

Why name the whole sequence. The agent is judged not on any single step but on the whole run. Naming it $\tau$ lets us talk about the total score of a run — which is exactly the next symbol.

11. Return $G_t$ and discount $\gamma$ — the total score

Why not just the immediate reward $r_t$? Because good moves can hurt now to help later (sacrifice a chess piece for checkmate). The agent must aim at the sum, not the flash. This is the single most common beginner error — see Policy and Value Functions and the parent's "Mistake 1".

Picture: a row of coins fading to grey as they go into the distance. The coin now is full-colour (${\gamma }^0=1$); the coin $k$ steps away is worth a fraction ${\gamma }^k$ of its face value.

$G_t=r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+\cdots ={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}$

Reading the $\sum$. The stretched-S $\sum$ means "add up." The little $k=0$ underneath and $\infty$ on top say "let $k$ run from $0$ upward forever, adding one term each time." Each term is reward $k$ steps ahead, faded by ${\gamma }^k$.

Why $\gamma <1$ keeps the sum finite. If every reward is at most $R_{\max }$ in size, then

$|G_t|\le {\sum }_{k=0}^{\infty }{\gamma }^k{R}_{\max }=R_{\max }\cdot \frac{1}{1-\gamma },$

using the fact that a shrinking geometric sum $1+\gamma +{\gamma }^2+\cdots$ settles down to $\frac{1}{1-\gamma }$. A finite total is what makes "maximize the return" a sensible goal. This trade-off — grab now vs. wait for more — is the heart of Exploration vs Exploitation and, in the simplest one-state case, Multi-Armed Bandits.

12. Expectation $\mathbb{E}$ — averaging over randomness

Picture: roll a die many times and take the mean of the numbers — that long-run mean is the expectation.

Why the topic needs it. Because both the policy and the world can be random, the return of a single run is luck. The agent wants the policy whose average return is highest: ${\pi }^{\ast }=\arg {\max }_{\pi }{\mathbb{E}}_{\tau \sim \pi }[G_0]$.

Prerequisite map

Read it upward: the humble time step feeds state/action/reward; those feed the policy and the world's physics; physics plus the Markov property give the MDP; the trajectory plus discounting give the return; averaging returns gives the goal.

Equipment checklist

Cover the right-hand side and see if you can answer each before revealing.