Visual walkthrough — Deep Q-Networks (DQN)

The result we are marching toward:

$\mathcal{L}(\theta )=(\underset{\text{target }y}{\underbrace{r+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime };{\theta }^{-})}}-Q(s,a;\theta ){)}^2$

Every letter in there is a promise. Let us earn each one.

---

Step 1 — What is a "state", an "action", and a "reward"?

WHAT. Picture an agent (a little robot) standing in a world. At each tick of the clock it sees a state $s$ (everything it can observe right now — a game screen, a position), chooses an action $a$ (press a button, step left), and the world hands back a reward $r$ (a number: +1 good, −1 bad, 0 nothing) and drops it into a new state $s^{\prime }$.

WHY these three. They are the only things the agent ever touches. Learning = discovering which $a$ in which $s$ leads to the most reward over time. Everything else is bookkeeping.

PICTURE. The loop below is the whole game: state → action → reward + next state → repeat.

---

Step 2 — Why we want ONE number per (state, action): the $Q$ value

WHAT. We invent a scorekeeper called ==$Q(s,a)$== — a single number meaning "how much total reward do I expect if I take action $a$ here, and play well forever after?" Big $Q$ = promising move.

WHY a total, not just the next reward. A move that gives $r=0$ now but sets up a jackpot later must still look good. So $Q$ must look into the future, not just the next step.

WHY "discounted". A reward now is worth more than the same reward in 100 turns (it's more certain, more useful). We shrink far-off rewards by a factor ==$\gamma$== (gamma, between 0 and 1) raised to how many steps away they are: ${\gamma }^0,{\gamma }^1,{\gamma }^2,\dots$ — a decaying staircase.

PICTURE. Future rewards stacked up, each shorter than the last because ${\gamma }^k$ shrinks.

$Q(s,a)=\underset{\text{now}}{\underbrace{r_t}}+\underset{\text{next, shrunk once}}{\underbrace{\gamma r_{t+1}}}+\underset{\text{shrunk twice}}{\underbrace{{\gamma }^2{r}_{t+2}}}+\cdots$

• $r_t$ — reward received this turn, full weight (${\gamma }^0=1$).
• $\gamma r_{t+1}$ — next turn's reward, worth a fraction $\gamma$.
• the "$\cdots$" — the infinite tail, each term smaller than the last. Because $0\le \gamma <1$ the total is a finite number even though the sum never ends.

This is the Q-Learning value; see also Temporal Difference Learning for the update flavour.

---

Step 3 — The recursion trick: fold the infinite tail into itself

WHAT. We split the giant sum into two pieces: the first reward, and everything after.

$Q(s,a)=r_t+\gamma (r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\cdots )$

WHY. Look inside the bracket — it is itself a $Q$-shaped sum, but starting one step later, from the next state $s^{\prime }$. So the tail equals the value of $s^{\prime }$. We just discovered the future is a smaller copy of the present. That self-reference is the recursion that makes learning possible.

WHY "max". From $s^{\prime }$ we get to choose again — and to play optimally we pick the best next action $a^{\prime }$. "Best" means the largest $Q$, written ==${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$==: scan every possible next action, keep the biggest score.

PICTURE. A tree: the current move, then the fan of next actions from $s^{\prime }$; the thick branch is the $\max$ (the one we'd actually take).

---

Step 4 — Why we can't just USE this equation (enter the neural net)

WHAT. Bellman defines $Q^{\ast }$ but doesn't hand it to us — the right side contains $Q^{\ast }$ itself, the very thing we don't know. And there are far too many states to store one number each (think every possible game screen).

WHY a network. Instead of a lookup table we use a neural network $Q(s,a;\theta )$: a machine with tunable knobs ==$\theta$== (theta, the weights) that eats a state and guesses a $Q$ value for each action. Similar states → similar guesses, so it generalises — learning about one screen teaches it about lookalike screens.

PICTURE. Left: an impossibly tall table (impossible). Right: a compact net turning a state into action-scores.

• $\theta$ — the adjustable weights. "Training" = nudging $\theta$ so the guesses get right.
• The net outputs one number per discrete action. (This is exactly why DQN needs discrete actions — you can't list a $\max$ over a continuous knob.)

---

Step 5 — Bootstrapping: build a target out of our own guess

WHAT. We can't compare our guess to the true $Q^{\ast }$ (unknown). So we compare it to the best estimate we can assemble right now from real data: the reward we actually saw, plus our own discounted guess of the next state's value. Call this the ==target $y$==:

$y=\underset{\text{real, observed}}{\underbrace{r}}+\underset{\text{our own guess of the future}}{\underbrace{\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime };{\theta }^{-})}}$

WHY it's legitimate. Half of $y$ is ground truth ($r$ really happened). Only the tail is a guess. Each update makes $y$ a little truer, which makes the next $y$ truer — the guess pulls itself up by its bootstraps. This is bootstrapping.

WHY ${\theta }^{-}$ and not $\theta$. The knobs used inside $y$ are a frozen copy ==${\theta }^{-}$== (the target network idea's ancestor). If $y$ used the same live $\theta$ we are editing, the target would jump the instant we take a step — a dog chasing its own tail. Freezing it gives a steady thing to aim at. We refresh ${\theta }^{-}\leftarrow \theta$ only every $C$ steps.

PICTURE. Two networks: the live one we train (blue), and its frozen older twin ${\theta }^{-}$ (yellow) that builds the target.

---

Step 6 — The TD error and the loss: measuring the gap

WHAT. The TD error ($\delta$, delta) is simply target minus current guess:

$\delta =y-Q(s,a;\theta )$

If $\delta >0$ our net underestimated — push $Q(s,a)$ up. If $\delta <0$ it overestimated — push down. If $\delta =0$, Bellman is satisfied here; leave it.

WHY square it. We want any gap punished (sign shouldn't matter) and big gaps punished more. Squaring does both and is smooth to differentiate:

$\mathcal{L}(\theta )=(y-Q(s,a;\theta ){)}^2={\delta }^2$

WHY freeze $y$ during the gradient. We differentiate $\mathcal{L}$ w.r.t. $\theta$ but treat $y$ as a constant label (stop-gradient). It contains ${\theta }^{-}$, not $\theta$, so there's nothing live to differentiate anyway — and pretending otherwise makes training diverge.

PICTURE. A parabola ${\delta }^2$: gradient descent slides the guess toward $\delta =0$ at the bottom.

$\theta \leftarrow \theta +\alpha \underset{\text{how wrong}}{\underbrace{\delta }}\underset{\text{which knobs move }Q}{\underbrace{{\nabla }_{\theta }Q(s,a;\theta )}}$

• $\alpha$ — step size (learning rate); how far each nudge goes.
• ${\nabla }_{\theta }Q$ — the direction in knob-space that raises $Q(s,a)$; multiply by $\delta$ to move the right way by the right amount.

---

Step 7 — The degenerate case: a terminal state

WHAT. When $s^{\prime }$ is terminal (the episode ended — game over, goal reached), there is no next state to evaluate. So the future term vanishes and the target collapses to just the reward:

$y=r(\text{terminal }{s}^{\prime })$

WHY. $\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$ asks "value of the best move from $s^{\prime }$" — but from a finished game there are no moves and no future reward. Keeping the term would invent phantom future value and bias every estimate upward. Dropping it is not optional; it's the boundary condition of the recursion.

PICTURE. Two paths side by side: non-terminal (future tail present) vs terminal (tail cut off).

---

Step 8 — Two worked targets, drawn on the number line

Non-terminal — $r=1$, $\gamma =0.9$, target net gives $Q(s^{\prime },\text{left})=2$, $Q(s^{\prime },\text{right})=5$; current $Q(s,a)=3$.

${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })=\max (2,5)=5,y=1+0.9\times 5=5.5,\delta =5.5-3=+2.5$

Positive $\delta$ → net underestimates → push $Q(s,a)$ up toward $5.5$.

Terminal — same numbers but $s^{\prime }$ is terminal, and old $Q(s,a)=1.2$.

$y=r=1,\delta =1-1.2=-0.2$

Negative $\delta$ → nudge $Q(s,a)$ down.

PICTURE. Both targets and current guesses plotted on one axis; arrows show which way $Q$ moves.

---

The one-picture summary

Everything on this page compressed: the game loop feeds transitions into the replay buffer; a random batch is drawn; the frozen twin ${\theta }^{-}$ builds the target $y$; we compare to the live guess, get $\delta$, and slide the knobs downhill on ${\delta }^2$. Every $C$ steps the twin is refreshed.

Recall Feynman retelling — the whole walkthrough in plain words

A robot plays a game and writes down each moment as (what I saw, what I did, the points I got, what I saw next). It wants a scorekeeper that, given a screen and a button, guesses the total future points. That total is a shrinking sum of rewards — and cleverly, the future is just a smaller copy of the present (recursion), so the score = points now + a shrunk copy of "the best I can do next". Since it doesn't know the true best, it uses its own guess as a stand-in (bootstrapping), but reads that guess off a frozen older twin so the target doesn't wobble. It measures how wrong the guess is (the TD gap), squares it so both over- and under-guesses hurt, and nudges its knobs downhill. When the game ends there's no "next", so the score is just the last reward. Repeat millions of times and the guesser becomes a champion.

---

Connections

• Deep Q-Networks (DQN) — the parent note this page zooms into.
• Bellman Equation — Steps 3 provides its picture.
• Q-Learning — the tabular version of Steps 2–3.
• Temporal Difference Learning — Step 6's $\delta$ is the TD error.
• Experience Replay — where the batches in the summary come from.
• Double DQN — refines the frozen-twin $\max$ of Step 5.
• Dueling DQN — an alternative network shape for Step 4.
• Epsilon-Greedy Exploration — how actions in Step 1 are actually chosen.
• Policy Gradient Methods — the sibling family that skips $Q$ entirely.