Reward hacking and specification gaming: Foundations

Before you can read the parent note Reward hacking and specification gaming, you must be able to read its symbols the way you read plain words. This page assumes you know nothing about reinforcement learning. Every letter below is earned before it is used.

1. The agent, the world, and one clock tick

Picture a robot in a room. At every moment it looks, then acts, then the world changes. That loop is the atom of everything that follows.

The figure below shows this loop as two boxes. On the left is the current state box (the robot's snapshot of the room). The red arrow in the middle is the action the robot chooses — that is the one thing the agent controls, so it is coloured red. On the right the world processes that action and hands back two things along the black return arrow: a reward number (a score for the move it just made) and the next state (the room's new freeze-frame), which becomes the new left-hand box on the following tick. Trace the red arrow first, then follow the black one back to see the loop close.

For now, read the figure with plain words only — snapshot in, action out, score and new snapshot back. Sections 2 and 3 will attach precise symbols ( $s_{t}$ , $a_{t}$ , $R_{t}$ ) to exactly these three plain-word roles, so you can return here and re-read the figure symbolically.

Figure — Reward hacking and specification gaming

Figure 1 — One tick of the agent–world loop: the current snapshot → red action the robot chooses → world update → a score and the next snapshot returned. (Symbols attached in Sections 2–3.)

Why do we need both letters? Because reward hacking is always a state–action story: the agent found some action in some state that scores well but betrays your intent. You cannot describe the betrayal without naming what was done and where.

2. A subscript is just a timestamp

The room's loop happens over and over. We label each turn of the loop with a small number written below the letter.

3. Reward $R$ — the score you wrote

Why a function and not just a number? Because the score depends on the situation. Winning a race scores high; crashing scores low — the same key ( $R$ ) reads different values for different $(s, a)$ inputs. The entire topic lives on the fact that you, the human, wrote this function, and you wrote it imperfectly.

4. True utility $U$ — the wish in your head

The figure below plots both scoreboards against a slice of the state–action space (horizontal axis = which situation you are in, vertical axis = the score). The black curve is the true utility $U$ : one honest hump, peaking in the middle where good behaviour lives. The red curve is the reward $R$ : it lies right on top of the black curve in the middle (they agree on the situations you imagined), but far to the right it grows a tall extra red spike where $R$ is huge while $U$ has already flattened to near zero. The dotted vertical lines mark each curve's peak location — notice the red flag and the black flag sit at different horizontal positions. That spike is exactly a reward hack: a situation the score loves and you do not.

Figure 2 — Two scoreboards over a slice of state–action space: black $U$ (true utility, one honest peak) vs red $R$ (reward, with an extra spike at the edge). Their peaks — the dotted lines — sit at different places.

The single most important fact of this topic: $R \neq = U$ . They usually agree on the situations you imagined while designing (that's why you thought $R$ was fine), but drift apart in situations you never pictured. The agent lives at the edges, on the red spike.

5. Reading "argmax" and "≈"

Three tiny bits of grammar the parent leans on:

That mismatch of flag-locations is reward hacking in one picture: the robot walks to the red flag, which is not where your black flag is.

6. Expectation $E$ and randomness $\sim$

The robot's future is not fixed — dice get rolled (the world is noisy, the policy explores). So we talk about averages.

7. Trajectory $τ$ and policy $π$

Definition Policy

$π$ — and why it can be random A policy $π$ is the agent's strategy: a rule that, for any state, decides which action to take. The fullest form is a distribution written $π (a ∣ s)$ — read "the probability the policy picks action $a$ when it sees state $s$ ."

Picture a lookup table where each state points not to a single arrow but to a spinner: it lists a probability for each possible action, and the agent spins to choose. A deterministic policy is just the special case where one action gets probability $1$ (the spinner always lands on the same arrow). Because $π$ can be a spinner, running it produces different trajectories on different runs — which is exactly why the previous section needs $τ \sim π$ ("roll a trajectory from the policy") and why we take an average $E_{τ \sim π}$ over those rolls.

8. Discount $γ$ , the sum $\sum$ , and how long an episode runs

Definition Episode length

$T$ vs infinite horizon $\infty$ Real games end: the boat crosses the line, the robot drops the ball, the round times out. The tick where a run stops is a terminal state, and $T$ is that final tick — the height of the filmstrip. A run that ends is a finite-horizon episode, written $\sum_{t = 0}^{T}$ .

Some settings have no natural end (a server that runs forever). We model those as infinite-horizon, writing the top of the sum as $\infty$ (the sideways-8 = "goes on without stopping"), i.e. $\sum_{t = 0}^{\infty}$ .

Picture two filmstrips: one with a clear last frame (finite, $T$ ), one that runs off the edge of the table forever ( $\infty$ ). The discount factor below is precisely the trick that lets the never-ending strip still have a finite total score — and a terminal state is just the point where all later rewards are $0$ , so the finite sum and the infinite sum agree.

Why do we need $γ$ ? Two reasons. Convergence: without shrinking, an infinite horizon sum of rewards could blow up to infinity and be meaningless; with $0 \leq γ < 1$ the fading terms add to a finite total even when $T = \infty$ . Impatience: sooner rewards should count more. This is exactly why the Tetris pause-forever hack in the parent is so revealing — pausing keeps the game alive infinitely (no terminal state ever reached), and if the reward is "survive", the tower of survival-points never falls.

9. Putting it together: the return $J$

Now every symbol in the parent's headline equation is earned. We write the infinite-horizon form; for a run that ends at terminal tick $T$ , just read every reward past $T$ as $0$ , and the sum stops there on its own.

Figure 3 — Each tick contributes a raw reward (white bars, all $= 1$ ) but only its discounted share (red bars, $γ^{t}$ ) counts toward the return $J$ . Far-future bars shrink toward zero — which is why an infinite run still totals a finite score.

10. The last two symbols: $λ$ and $ϵ$

Prerequisite map

These foundations feed directly into deeper topics: once you see $R \neq = U$ , you can study why agents drift when the world changes (robustness and distribution shift), why they pick up sneaky sub-goals (instrumental convergence), how a learned inner optimiser can have its own hidden $U$ (mesa-optimization), how we might learn the black scoreboard instead of guessing it (value learning), and how we peer inside to catch the hack (interpretability).

Equipment checklist

Cover the right side and answer aloud. If you stumble, reread that section.

What does a state

s

picture as, and what is

S

One freeze-frame of the world;

S

is the box of all possible frames.

s_{t}

ever "s times t"?

No — the subscript

t

is just a timestamp / frame number, never multiplication.

In words, what is

R (s, a)

A function: give it a state and an action, it returns one number, the score for that move (the red scoreboard).

How does

R_{t}

relate to

R (s, a)

R_{t} = R (s_{t}, a_{t})

— the same scoreboard function evaluated at this tick's state and action;

R

is the rule,

R_{t}

is this turn's reading.

What is

U

, and what is the single most important fact about

R

and

U

True utility, the wish in your head; the key fact is

R \neq = U

— they agree on imagined cases, drift apart elsewhere.

What does

ar g max_{x} f (x)

give you — a value or a location?

The location (the input

x

) where

f

peaks, not the peak's height.

Read

E_{τ \sim π} [\cdot]

aloud.

"The long-run average, over playthroughs

τ

rolled from strategy

π

" — the subscript names which random source we average over.

Is a policy

π

always a fixed rule?

No — in general it is a distribution

π (a ∣ s)

, a spinner over actions; a deterministic rule is the special case where one action has probability

1

What is a terminal state, and what is

T

The tick where a run stops;

T

is that final tick — the episode length. A finite run sums to

T

; past a terminal state all rewards are

0

Why does the return use

γ^{t}

inside the sum?

To shrink far-future rewards so even an infinite-horizon sum stays finite and sooner rewards count more.

Write

J_{true} (π)

and say how it differs from

J (π)

J_{true} (π) = E_{τ \sim π} [\sum γ^{t} U (s_{t}, a_{t})]

— the same discounted tower built on true utility

U

instead of reward

R

What is

Δ J = J (π) - J_{true} (π)

in one sentence?

The height gap between the red-scoreboard tower and the black-scoreboard tower — big and positive means reward hacking.

R^{'} = R - λ AUX

, what are

AUX

and

λ

AUX

is a "weirdness meter" penalising unusual side-effects;

λ

is the dial setting how hard to punish it.