Visual walkthrough — Target networks

Everything below is built from scratch. If you have never seen a Q-value, a gradient, or the letter $\theta$, start at Step 1 and don't skip.

---

Step 1 — What is $Q$, really? A number pinned to a picture

WHAT. We have an agent (a little robot) sitting in a state $s$ — think "which square of a grid I'm standing on". From that square it can take an action $a$ — "go left", "go right". A number called $Q(s,a)$ is our guess of how good it is to take action $a$ from state $s$, counting all future reward.

WHY this number. To act well the robot must compare actions. If $Q(s,\text{left})=2$ and $Q(s,\text{right})=5$, "right" looks better. That single number is the whole basis of the decision.

PICTURE. Look at the figure: three squares (states), and from the middle square two arrows for two actions. Each arrow carries its $Q$ number. The taller the bar, the more promising the action.

---

Step 2 — Where should $Q$ point? The Bellman target

WHAT. After taking $a$, the robot gets an immediate reward $r$ (a small payout) and lands in a next state $s^{\prime }$. From $s^{\prime }$ the best it can do is the largest $Q$ over next actions, written ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$. The Bellman rule says a correct $Q$ must satisfy:

$y=\underset{\text{cash now}}{\underbrace{r}}+\underset{\text{how much we care about later}}{\underbrace{\gamma }}\underset{\text{best future value from }{s}^{\prime }}{\underbrace{\underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })}}$

WHY these pieces. $r$ is what you actually earned this step. But good moves also set up good futures, so we add the best value reachable next. $\gamma$ (a number between $0$ and $1$, e.g. $0.9$) shrinks future reward a little — a dollar tomorrow is worth slightly less than a dollar today. We use $\max$ because the robot will later act greedily, so we assume the best next move.

PICTURE. The figure shows the "one-step lookahead": current square → reward arrow → next square → the tallest next-action bar. Stack them and you get $y$, the number $Q(s,a)$ should equal.

---

Step 3 — Turning "should equal" into a loss to minimise

WHAT. Right now $Q(s,a;\theta )$ (our prediction) probably does not equal $y$ (our target). The gap between them is the error. To measure it fairly (positive or negative gaps both count) we square it:

$L(\theta )=(\underset{\text{where we want to be}}{\underbrace{y}}-\underset{\text{where we are}}{\underbrace{Q(s,a;\theta )}}{)}^2$

WHY squared. Squaring makes the error always positive and punishes big misses more than small ones. The bottom of this bowl-shaped curve is where prediction = target. Training means rolling downhill.

WHY this tool (gradient). To roll downhill we need the gradient ${\nabla }_{\theta }L$ — the arrow pointing in the direction $\theta$ should move to increase $L$. We step the opposite way to decrease it. This is the only tool that tells us which way to nudge thousands of knobs at once.

PICTURE. The figure draws $L$ as a parabola in the prediction value: the target $y$ sits at the bottom; today's prediction sits up the wall; the arrow shows the downhill nudge.

---

Step 4 — The trap: the target moves when you shoot

WHAT. Here is the subtle disaster. The target is $y=r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };\theta )$ — it is built from the same knobs $\theta$ as the prediction. So when a gradient step changes $\theta$ to lower $Q(s,a;\theta )$'s error, it also changes ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };\theta )$ — the target itself shifts.

WHY it hurts. You aim, you shoot, and the bullseye slides — often in the same direction, because both sit on the shared knobs. The error $y-Q$ may refuse to shrink; the update is not guaranteed to contract the gap. Combined with bootstrapping and off-policy data (the "Deadly Triad"), this can oscillate or diverge.

PICTURE. The figure animates the tail-chase: the archer (prediction) steps toward the target, but an arrow shows the target being dragged along by the very same step. The gap barely closes.

---

Step 5 — The fix: a second, frozen set of knobs ${\theta }^{-}$

WHAT. Keep two copies of the network:

• Online network $Q(s,a;\theta )$ — the learner, nudged every step.
• Target network $Q(s,a;{\theta }^{-})$ — a lagged copy, used only to build the target.

The target now reads:

$y=r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };\underset{\text{frozen knobs}}{\underbrace{{\theta }^{-}}})$

and we do not let gradients pass through ${\theta }^{-}$ — we treat it as a plain constant. So:

${\nabla }_{\theta }L=-2(\underset{\text{constant now}}{\underbrace{y}}-\underset{\text{only thing that moves}}{\underbrace{Q(s,a;\theta )}}){\nabla }_{\theta }Q(s,a;\theta )$

WHY it works. Because $y$ is built from frozen ${\theta }^{-}$, it does not move when we nudge $\theta$. The bullseye is nailed down. Locally the problem becomes ordinary supervised regression toward a fixed label — and that is stable.

PICTURE. Two boxes: the online box glowing (updating), the target box greyed/frozen. A dashed one-way arrow shows ${\theta }^{-}$ feeding the target $y$, and a red "no-gradient" barrier blocking backflow into ${\theta }^{-}$.

---

Step 6 — Two ways to refresh the frozen copy

WHAT. A permanently frozen ${\theta }^{-}$ would learn nothing new. So we refresh it, but slowly. Two recipes:

Hard update (periodic copy) — original DQN: ${\theta }^{-}\leftarrow \theta \text{every }C\text{ steps}$ Snap the copy over every $C$ steps; freeze in between.

Soft update (Polyak averaging) — DDPG / TD3 / SAC: ${\theta }^{-}\leftarrow \underset{\text{tiny}}{\underbrace{\tau }}\theta +\underset{\text{almost 1}}{\underbrace{(1-\tau )}}{\theta }^{-},0<\tau \ll 1$ Every step, drag ${\theta }^{-}$ a hair's-breadth toward $\theta$.

WHY two shapes. Hard update is dead simple (a big jump now and then). Soft update spreads the same motion across every step, so ${\theta }^{-}$ never jumps — it glides. Both keep the target slow.

PICTURE. The figure plots the online knob wandering (jagged plum line) and, over it, the two target behaviours: the hard copy as a staircase that jumps every $C$ steps, the soft copy as a smooth lagging curve (an exponential moving average).

---

Step 7 — Why soft update is an exponential memory

WHAT. Unroll the soft rule assuming $\theta$ barely changes. After $k$ steps, an old target value keeps weight $(1-\tau {)}^k$, which shrinks toward zero. So ${\theta }^{-}$ remembers roughly the last $\sim 1/\tau$ values of $\theta$.

WHY it matters. With $\tau =0.005$, the target has an effective memory of about $200$ steps — the online net can jitter step-to-step while the target sees only a smoothed average. That smoothing is the stability. Two limits sanity-check it:

• $\tau \to 1$: the sum collapses to just $\theta$ — target = online, no target network at all (back to Step 4's trap).
• $\tau \to 0$: nothing ever updates — the target is frozen forever.

PICTURE. The figure draws the decaying weights $(1-\tau {)}^k$ as shrinking bars — recent $\theta$ counts a lot, old $\theta$ fades — with $1/\tau$ marked as the "memory length".

---

Step 8 — Edge cases you must never miss

WHAT. Two boundary situations change the formula:

1. Terminal state. If $s^{\prime }$ ends the episode, there is no future, so the target drops its bootstrap term entirely: $y=r(\text{no }\gamma \max Q\text{ term})$

2. Double DQN correction. The plain $\max$ tends to overestimate (it always grabs the luckiest noisy value). Double DQN splits the job: the online net chooses the action, the target net scores it: $y=r+\gamma Q(s^{\prime },\underset{\text{online picks}}{\underbrace{\arg \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime };\theta )}};\underset{\text{target scores}}{\underbrace{{\theta }^{-}}})$

WHY. Forgetting the terminal mask injects imaginary future reward — a classic silent bug. And plain $\max$ bias and target-instability are different diseases; the target network cures instability, Double DQN cures overestimation. They stack.

PICTURE. The figure splits into two panels: left, a terminal square with a "wall" so no next-value flows ($y=r$); right, the Double-DQN handoff — online net's finger points at an action, target net stamps its value on it.

---

Worked numbers (matching the parent)

---

The one-picture summary

Everything compresses into this: a fast online loop learning against a slow, frozen target that is only occasionally (hard) or gently (soft) refreshed — the archer hitting a still bullseye.

Recall Feynman: the whole walkthrough in plain words

We taught a robot to score its moves with a number $Q$. The "right" score comes from the Bellman rule: cash now plus the best value later. But that right answer is computed from the same brain we're training — so every time we adjust the brain, the answer we're chasing also shifts. It's archery where your own shot drags the target. Solution: keep a second, older copy of the brain that we don't touch, and let it define the target. Now the bullseye holds still, so practice actually improves aim — ordinary regression, nice and stable. Every so often we copy the fresh brain into the old one (a big snap = hard update, or a tiny nudge every step = soft update). Tiny nudges give the old brain a long, smoothed memory, so the target glides instead of jumping. Two special cases: at the end of an episode there's no future, so the target is just the cash ($y=r$); and if we want to stop the $\max$ from being an over-optimist, we let the new brain pick the move and the old brain grade it (Double DQN). Freeze, regress, no backflow, then gently copy — FRO-ZEN.

---

Connections

• Deep Q-Networks (DQN) — home of the hard target update.
• Experience Replay — the other DQN stabiliser.
• Bellman Equation — supplies the target $y$ we freeze.
• Double DQN — the overestimation fix stacked on top.
• DDPG / TD3 / SAC — use soft (Polyak) target updates.
• Deadly Triad — the instability target nets tame.
• Two-Timescale Stochastic Approximation — the convergence theory.

Concept Map