Agent, environment, state, action, reward

Overview

The agent-environment interaction loop is the foundation of all reinforcement learning. Unlike supervised learning where we're told what the right answer is, RL agents must discover good behavior by trial and error, receiving only feedback (rewards) about how good their actions were.

The Five Components

Key insight: The agent contains the "brain" — the policy (action-choosing strategy) and possibly a model of how the world works.

Key insight: The environment boundary is a design choice. For a robot, we might include the robot's body in the environment and only the control algorithm as the agent. Or we might put motor controllers in the agent and only the external world in the environment.

Why this matters: In a fully observable environment (like chess), the agent sees the true state. In partially observable environments (like poker or a robot with limited sensors), the agent must make decisions from incomplete information.

Why actions are first-class: The agent controls ONLY its actions. It cannot directly control the state or reward — those emerge from the interaction.

Critical insight: Rewards are the ONLY way to communicate the goal. If you want an RL agent to walk forward, reward forward distance. If you accidentally reward not falling, it might learn to lie down!

The Interaction Loop: Deriving the Dynamics

Let's formalize how these components interact over time.

Step 1: Initial state At $t=0$, the environment is in state $s_0$ (drawn from some initial distribution $p(s_0)$).

Step 2: Agent observes The agent receives observation $o_0$ (in fully observable case, $o_0 = s_0$).

Step 3: Agent acts The agent's policy $\pi$ maps states/observations to actions. This can be:

• Deterministic: $a_t = \pi(s_t)$ — same state always gives same action
• Stochastic: $a_t \sim \pi(a|s_t)$ — sample action from probability distribution

Step 4: Environment responds The environment receives action $a_0$ and:

1. Transitions to new state: $s_1 \sim P(s'|s_0, a_0)$ where $P$ is the state transition probability
2. Emits reward: $r_0 = R(s_0, a_0, s_1)$ or $r_0 \sim R(s_0, a_0)$ if stochastic

Step 5: Loop repeats This produces a trajectory (also called episode or rollout): $s_0, a_0, r_0, s_1, a_1, r_1, s_2, a_2, r_2, \ldots$

Why this formulation? This is a Markov Decision Process (MDP). The "Markov" part means: given the current state $s_t$, the future is independent of the past. Formally: $P(s_{t+1}, r_t | s_t, a_t, s_{t-1}, a_{t-1}, \ldots) = P(s_{t+1}, r_t | s_t, a_t)$

This lets us make decisions based only on the current state, not the entire history.

Finite horizon (episode ends at time $T$): $G_t = r_t + r_{t+1} + r_{t+2} + \cdots + r_T = \sum_{k=0}^{T-t} r_{t+k}$

Why? Simple sum works when episodes have a clear end (game over, task completed).

Infinite horizon with discounting: $G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots = \sum_{k=0}^{\infty} \gamma^k r_{t+k}$

where $\gamma \in [0, 1)$ is the discount factor.

Why discount?

1. Mathematical: Makes the infinite sum converge (if rewards are bounded)
2. Practical: Reflects that near-term rewards are more certain/valuable than distant ones
3. Control: $\gamma$ close to 0 = myopic (greedy), $\gamma$ close to 1 = far-sighted

Derivation of why this converges: Assume rewards are bounded: $|r_t| \leq R_{max}$ for all $t$. $|G_t| = \left|\sum_{k=0}^{\infty} \gamma^k r_{t+k}\right| \leq \sum_{k=0}^{\infty} \gamma^k R_{max} = R_{max} \sum_{k=0}^{\infty} \gamma^k$

The geometric series sums to: $\sum_{k=0}^{\infty} \gamma^k = \frac{1}{1-\gamma} \quad \text{for } |\gamma| < 1$

Therefore: $|G_t| \leq \frac{R_{max}}{1-\gamma}$ which is finite. ✓

Recursive form (bootstrapping): $G_t = r_t + \gamma(r_{t+1} + \gamma r_{t+2} + \cdots) = r_t + \gamma G_{t+1}$

This recursion is the seed of the Bellman equation.

Trajectory example:

t=0: s₀=(1,1), agent observes (1,1), chooses a₀=North
     Environment: valid move, s₁=(1,2), r₀=-1

t=1: s₁=(1,2), agent chooses a₁=East
     Environment: valid move, s₂=(2,2), r₁=-1

t=2: s₂=(2,2), agent chooses a₂=North  
     Environment: hits wall, s₃=(2,2), r₂=-5

t=3: s₃=(2,2), agent chooses a₃=East
     Environment: valid move, s₄=(3,2), r₃=-1

... continues until reaching goal at (5,5)

t=12: s₁₂=(5,5), r₁₂=+10, episode ends

Return calculation (with $\gamma=0.9$): $G_0 = -1 + 0.9(-1) + 0.9^2(-5) + \cdots + 0.9^{12}(+10)$

Why the step penalty? Without it ($r=0$ for non-goal states), the agent has no incentive to reach the goal quickly. It could wander forever and never learn a good policy.

Why this reward structure?

• First term: Negative squared error → penalizes deviation from target (comfortable temperature)
• Second term: Small penalty on energy use → encourages efficiency
• Both terms are negative → maximizing reward means minimizing deviations and energy

Trajectory step:

t=0: s₀=(18°C, 10°C), agent chooses a₀=0.8 (80% heater)
     Environment physics: Q = heater_power × max_heating
                Heat_loss = k × (T_inside - T_outside)
                T_new = T_old + (Q - Heat_loss) × dt
     Result: s₁=(19.5°C, 10°C), r₀=-(19.5-21)² - 0.01×0.8² = -2.256

Why continuous actions? Temperature control naturally operates on a continuous scale. Discretizing to {off, low, medium, high} would lose precision.

Common Mistakes

Why it feels right: Rewards tell us what's good, so maximizing each reward seems logical.

The fix: The agent must maximize cumulative discounted reward (return), not immediate reward. Example: In chess, sacrificing a piece (negative immediate reward) to set up checkmate (large future reward) is optimal.

Formal: The agent learns a policy $\pi^*$ such that: $\pi^* = \arg\max_\pi \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \right]$ Not: $\pi = \arg\max_\pi \mathbb{E}[r_t]$ at each $t$ independently.

Why it feels right: Losses and rewards both measure performance, and we're used to minimization from ML.

The fix: RL rewards are designed to be maximized. If your task naturally has a loss (tracking error, energy waste), define $r = -\text{loss}$ so that maximizing reward means minimizing loss.

Example: For distance from goal $d$, use $r = -d$ not $r = d$.

Why it feels right: Simpler states = smaller state space = faster learning.

The fix: If you omit information needed to predict the future, you break the Markov property. The agent will see the same state but experience different outcomes depending on hidden history, making learning impossible.

Example: In Pong, a single frame doesn't show ball velocity. You need 2+ frames or include velocity in the state explicitly. Otherwise, the agent can't predict where the ball will go.

Or picture a feedback loop: Agent → Action → Environment → State+Reward → Agent (closes the loop)

You (the agent) can see the screen (the state) — where you are, where enemies are. You choose which button to press (action). The game world (environment) updates based on your button: your character moves, enemies move, and it decides how many points you earned.

At first, you try random buttons. But over time, you notice patterns: "When I'm near a cliff and press right, I fall and lose points. When I jump over the cliff instead, I get across safely." You're learning which actions work in which situations. That's exactly what RL agents do, except with math instead of button mashing!

The "return" is like your total score for the whole game. You don't just want to grab the nearest coin (immediate reward) — sometimes it's smarter to skip a coin to avoid danger, getting a better total score at the end.

Connections

• Markov Decision Process (MDP) — formal mathematical framework
• Policy and Value Functions — how agents encode their strategy
• Bellman Equations — recursive relationship for optimal returns
• Exploration vs Exploitation — how agents balance trying new actions vs using known good ones
• Partial Observability (POMDP) — when agent doesn't see full state
• Multi-Armed Bandits — simplified RL with only one state
• Model-Free vs Model-Based RL — whether agent learns environment dynamics

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)