Target networks: Worked examples — AI-ML deep dive

Prerequisite reading if any symbol below feels new: Deep Q-Networks (DQN), Bellman Equation, and the parent Target networks note.

First, the symbols — earned before use

Every example uses the same handful of quantities. Let's name them once, in plain words.

First, two words the whole page rests on: a state $s$ is "the situation the agent is in right now" (where the robot stands, what the game screen shows), and an action $a$ is "one of the moves the agent can choose" (go left, fire, move-to-shelf). A prime, as in $s^{'}$ and $a^{'}$ , just means "the next one" — the state you land in and an action you could take there.

Definition The cast of characters

$s$ — the state (current situation); $s^{'}$ — the next state after acting.
$a$ — the action taken; $a^{'}$ — a candidate next action.
$r$ — the reward, a single number the environment hands you after acting. Can be positive, negative, or zero.
$γ$ (gamma) — the discount factor, a knob between $0$ and $1$ saying "how much do I care about the future?". $γ = 0.9$ means a reward one step later is worth $90%$ of an immediate one.
$Q (s, a)$ — the network's guess of total future reward if you take action $a$ in state $s$ . It's just a number the network outputs.
$θ$ (theta) — the online network's dials (its weights). Changes every step.
$θ^{-}$ (theta-minus) — the target network's dials: a lagged snapshot of $θ$ .
$τ$ (tau) — the soft-update coefficient (used only in Ex 5–6): a tiny number between $0$ and $1$ saying "what fraction of the online net do I blend into the target each step?". Small $τ$ (e.g. $0.01$ ) = the target creeps slowly.
$α$ (alpha) — the learning rate (used in Ex 7): a small positive number saying "how big a step do I take toward the target each update?". Big $α$ = bold jumps, small $α$ = timid nudges.
$y$ — the target (the label we regress toward), built from $r$ , $γ$ , and the frozen $θ^{-}$ .

The symbol $max_{a^{'}}$ means "look at $Q (s^{'}, a^{'}; θ^{-})$ for every possible next action $a^{'}$ and keep the biggest number." Picture a row of bars — one per action — and you point at the tallest.

The scenario matrix

Below is every distinct cell a target-network calculation can land in. Each worked example is tagged with the cell(s) it covers.

#	Cell class	What's special	Covered by
C1	Non-terminal, positive reward, hard update	the "vanilla" backup	Ex 1
C2	Non-terminal, negative reward	sign of $r$ flips the target down	Ex 2
C3	Terminal state	future term dropped, $y = r$	Ex 3
C4	Zero / degenerate: $r = 0$ and $γ = 0$	pure regression, no bootstrap	Ex 4
C5	Soft update (Polyak), one step	EMA arithmetic, tracking speed	Ex 5
C6	Limiting $τ$ : $τ \to 1$ and $τ \to 0$	recovers "no target net" / "frozen forever"	Ex 6
C7	Divergence demo: coupled vs frozen	why freezing helps, numerically	Ex 7
C8	Double DQN twist	online selects, target evaluates	Ex 8
C9	Real-world word problem	full pipeline end-to-end	Ex 9
C10	Exam twist: negative $Q$ values in the max	tallest bar can be negative	Ex 10

Figure 1 lays out the matrix as a grid. The horizontal axis is what kind of transition you're backing up (positive-reward, negative-reward, terminal, degenerate); the vertical axis is how the target network is updated (hard copy, soft Polyak, limiting/theory). Each coloured pin is one worked example dropped into the cell it exercises — the legend on the figure maps every colour to its example number, so you can see at a glance that the ten examples together fill every cell.

Worked examples

Worked example Ex 1 — C1: Non-terminal, positive reward, hard update

$r = 1$ , $γ = 0.9$ . The frozen target network outputs over 3 actions $Q (s^{'}, \cdot; θ^{-}) = [2.0, 5.0, 3.0]$ . Online net currently says $Q (s, a; θ) = 4.0$ .

Forecast: which action's value gets used, and is the TD error positive or negative? Guess before reading.

Take the max over next actions. $max_{a^{'}} Q (s^{'}, a^{'}; θ^{-}) = max (2.0, 5.0, 3.0) = 5.0$ . Why this step? The greedy Bellman backup asks "best I can do next" — the tallest bar, from the frozen net so the label can't wobble.
Build the target. $y = 1 + 0.9 \cdot 5.0 = 1 + 4.5 = 5.5$ . Why this step? Add today's reward to the discounted best future — this is the label the online net will chase.
TD error. $δ = y - Q (s, a; θ) = 5.5 - 4.0 = 1.5$ . Why this step? Positive $δ$ means the online guess is too low; gradient descent nudges it up.

Verify: $1 + 0.9 (5.0) = 5.5$ ✓; error $5.5 - 4.0 = 1.5 > 0$ , so online value rises toward $5.5$ , sanity-consistent. $θ^{-}$ (hence $5.5$ ) is untouched this step.

Worked example Ex 2 — C2: Non-terminal,

negative reward $r = - 2$ , $γ = 0.95$ . Frozen target outputs $Q (s^{'}, \cdot; θ^{-}) = [1.0, 0.5, - 0.4]$ . Online $Q (s, a; θ) = 0.0$ .

Forecast: a penalty of $- 2$ — will the target be negative overall? Guess the sign of $y$ .

Max over next actions. $max (1.0, 0.5, - 0.4) = 1.0$ . Why this step? Even with a bad reward now, we still pick the best future — the max ignores the reward, it acts only on $θ^{-}$ 's outputs.
Target. $y = - 2 + 0.95 \cdot 1.0 = - 2 + 0.95 = - 1.05$ . Why this step? The negative reward dominates the small discounted future → the label is negative, telling the net "this state-action is bad."
TD error. $δ = - 1.05 - 0.0 = - 1.05$ . Why this step? Negative $δ$ means online value is too high; descent pushes it down.

Verify: $- 2 + 0.95 (1.0) = - 1.05$ ✓; $δ = - 1.05 < 0$ so value decreases — correct direction for a penalising transition.

Worked example Ex 3 — C3:

Terminal state $r = 3$ , $γ = 0.99$ . The next state $s^{'}$ is terminal (game over). The target net would output $[8.0, 9.0, 7.0]$ if queried. Online $Q (s, a; θ) = 6.0$ .

Forecast: does the $9.0$ enter the target? Guess $y$ .

Apply the terminal mask. Because $s^{'}$ is terminal, there is no successor — drop the whole $γ max Q$ term. Why this step? The Bellman backup has nothing to bootstrap from; querying the frozen net here injects phantom future value (a classic bug).
Target. $y = r = 3$ . Why this step? All future reward is exactly zero after the episode ends, so the label is purely the final reward.
TD error. $δ = 3 - 6.0 = - 3.0$ . Why this step? The online net over-valued a dead-end state; descent corrects it down.

Verify: terminal ⇒ $y = r = 3$ ; the $9.0$ is ignored ✓. If we had (wrongly) bootstrapped, $y = 3 + 0.99 (9) = 11.91$ — a $\sim 4 \times$ inflation, showing why the mask matters.

Worked example Ex 4 — C4: Zero / degenerate (

$r = 0$ , $γ = 0$ ) $r = 0$ , $γ = 0$ . Frozen target outputs $[100, - 5, 42]$ (deliberately wild). Online $Q (s, a; θ) = 2.5$ .

Forecast: with $γ = 0$ , does anything from the future survive? Guess $y$ .

Discount kills the future. $γ max Q = 0 \cdot 100 = 0$ . Why this step? $γ = 0$ means "I care nothing about the future" — the max is multiplied away entirely, no matter how large.
Target. $y = 0 + 0 = 0$ . Why this step? With no reward and no discounted future, the label is exactly zero — this is plain supervised regression toward 0, no bootstrapping at all.
TD error. $δ = 0 - 2.5 = - 2.5$ . Why this step? Negative $δ$ says the online net is over-valued relative to the true label $0$ ; descent pushes it down toward zero.

Verify: $0 + 0 \cdot 100 = 0$ ✓; $δ = 0 - 2.5 = - 2.5 < 0$ . This is the degenerate limit where target networks become irrelevant — no bootstrap means no moving-goalpost, so freezing changes nothing.

Worked example Ex 6 — C6: Limiting

$τ$ ( $τ \to 1$ and $τ \to 0$ ) Same weights as Ex 5: $θ = 10$ , $θ^{-} = 4$ . Evaluate the soft update at the two extremes.

Forecast: which extreme means "no target network at all"? Which means "frozen forever"?

$τ = 1$ . $θ_{new}^{-} = 1 (10) + 0 (4) = 10$ . Why this step? The target instantly becomes the online net — there is effectively no lag, no target network. Learning reverts to the unstable coupled scheme.
$τ = 0$ . $θ_{new}^{-} = 0 (10) + 1 (4) = 4$ . Why this step? The target never moves — frozen forever. Stable but the label eventually goes stale and learning stalls.
Interpret the middle. Effective memory $\approx 1/ τ$ steps; $τ = 0.01 \Rightarrow\sim 100$ steps of "memory." Why this step? This is the sweet spot from Ex 5 — slow enough to be stable, fresh enough to keep learning.

Verify: $τ = 1 \Rightarrow θ^{-} = 10$ (=online) ✓; $τ = 0 \Rightarrow θ^{-} = 4$ (unchanged) ✓; $1/0.01 = 100$ ✓.

Worked example Ex 7 — C7: Divergence demo (coupled vs frozen)

A stripped-down 1-D toy. One weight $w$ predicts $\overset{q}{^} = w$ . The "next-state" value is $2 w$ (a self-referential feature that mimics bootstrapping), reward $r = 0$ , $γ = 1$ , learning rate $α = 0.5$ , start $w_{0} = 1$ .

Update rule: $w_{t + 1} = w_{t} + α (y_{t} - \overset{q}{^}_{t})$ where $y_{t} = r + γ \cdot (2 w_{target})$ .

Forecast: with a shared weight ( $w_{target} = w_{t}$ ) will $w$ settle or blow up? Guess.

Coupled (no target net), step 1. $y_{0} = 0 + 1 \cdot 2 (1) = 2$ ; $\overset{q}{^}_{0} = 1$ ; $w_{1} = 1 + 0.5 (2 - 1) = 1.5$ . Why this step? The label $y$ grows with $w$ — chasing it makes the next label grow more.
Coupled, step 2. $y_{1} = 2 (1.5) = 3$ ; $\overset{q}{^}_{1} = 1.5$ ; $w_{2} = 1.5 + 0.5 (3 - 1.5) = 2.25$ . Why this step? $1 \to 1.5 \to 2.25$ : each step multiplies $w$ by $1.5$ — this is geometric divergence.
Frozen target, step 1. Freeze $w_{target} = 1$ , so $y = 2 (1) = 2$ stays fixed. $w_{1} = 1 + 0.5 (2 - 1) = 1.5$ ; step 2 with the same frozen $y = 2$ : $w_{2} = 1.5 + 0.5 (2 - 1.5) = 1.75$ . Why this step? Now the label is a constant $2$ ; $w$ converges to $2$ (a stationary regression), no blow-up.

Verify: coupled: $w_{2} = 2.25 > w_{1} = 1.5 > w_{0} = 1$ (growing) ✓; frozen: $w_{2} = 1.75$ sits between $1.5$ and the target $2$ , moving toward it ✓. Freezing turned divergence into convergence.

Worked example Ex 8 — C8: Double DQN twist

$r = 0.5$ , $γ = 0.9$ . Online net: $Q (s^{'}, \cdot; θ) = [3.0, 1.0, 2.0]$ . Target net: $Q (s^{'}, \cdot; θ^{-}) = [2.2, 1.5, 2.8]$ .

Forecast: vanilla DQN uses the target net's max $(2.8)$ . Double DQN uses the online argmax to index the target net. Which value gets evaluated? Guess.

Online net selects the action. $ar g max_{a^{'}} Q (s^{'}, a^{'}; θ) =$ action $0$ (value $3.0$ is largest among online). Why this step? Decoupling: the net that chooses is not the net that scores, which cancels the optimistic bias of a single max.
Target net evaluates that chosen action. Take target value at index $0$ : $Q (s^{'}, a^{⋆}; θ^{-}) = 2.2$ . Why this step? We trust the frozen net's number but the online net's choice — this is the whole Double DQN idea.
Target. $y = 0.5 + 0.9 \cdot 2.2 = 0.5 + 1.98 = 2.48$ . Why this step? Compare with vanilla DQN: $y_{vanilla} = 0.5 + 0.9 (2.8) = 3.02$ . Double DQN gives a lower, less over-optimistic target.

Verify: online argmax $= 0$ ✓; Double target $0.5 + 0.9 (2.2) = 2.48$ ✓; vanilla $0.5 + 0.9 (2.8) = 3.02$ ✓; $2.48 < 3.02$ so overestimation is reduced ✓. See Double DQN.

Worked example Ex 9 — C9: Real-world word problem (end-to-end)

A warehouse robot in state $s$ picks "move-to-shelf." It receives $r = - 0.1$ (energy cost), lands in non-terminal $s^{'}$ . $γ = 0.98$ . The target network scores the 4 actions at $s^{'}$ as $[0.9, 1.4, 1.2, 0.6]$ . The online net currently values the taken action at $Q (s, a; θ) = 1.0$ . Using hard-update vanilla DQN, compute the label and TD error, then say which way the online value moves.

Forecast: small negative reward but a decent future — will $y$ end up above or below $1.0$ ? Guess.

Best future (frozen). $max (0.9, 1.4, 1.2, 0.6) = 1.4$ . Why this step? The robot's future value = the best it can do at $s^{'}$ , read from the frozen target net.
Target. $y = - 0.1 + 0.98 \cdot 1.4 = - 0.1 + 1.372 = 1.272$ . Why this step? Tiny energy penalty barely dents a strong future, so the label lands above the current guess.
TD error & direction. $δ = 1.272 - 1.0 = 0.272 > 0$ → online value moves up. Why this step? The action was under-valued; the robot learns "move-to-shelf" is worth more than it thought.

Verify: $- 0.1 + 0.98 (1.4) = 1.272$ ✓; $δ = 1.272 - 1.0 = 0.272 > 0$ ✓, value rises. Units: all quantities are "reward units," dimensionally consistent — no unit clash.

Worked example Ex 10 — C10: Exam twist — negative

$Q$ values in the max $r = 0$ , $γ = 0.9$ . Frozen target outputs all negative: $Q (s^{'}, \cdot; θ^{-}) = [- 5.0, - 1.0, - 8.0]$ . Online $Q (s, a; θ) = - 3.0$ .

Forecast: the "max" of negatives — is it the number closest to zero or farthest? Guess.

Max of negatives. $max (- 5.0, - 1.0, - 8.0) = - 1.0$ . Why this step? "Biggest" means closest to zero here — the tallest bar even when all bars point down. A common trap is taking $- 8.0$ (the biggest magnitude).
Target. $y = 0 + 0.9 (- 1.0) = - 0.9$ . Why this step? Discounting a negative future keeps it negative but shrinks its size toward zero.
TD error. $δ = - 0.9 - (- 3.0) = - 0.9 + 3.0 = 2.1 > 0$ . Why this step? Online was more pessimistic than the target; value moves up toward $- 0.9$ .

Verify: $max (- 5, - 1, - 8) = - 1$ ✓; $y = 0.9 (- 1) = - 0.9$ ✓; $δ = - 0.9 - (- 3.0) = 2.1 > 0$ ✓.

Recall Quick self-test (cover the right side)

Vanilla target, $r = 1, γ = 0.9$ , frozen $Q (s^{'}, \cdot) = [2, 5, 3]$ , so $y = ?$ ::: $1 + 0.9 (5) = 5.5$ Same but $s^{'}$ terminal ::: $y = r = 1$ Soft update $τ = 0.01, θ = 10, θ^{-} = 4$ gives ::: $4.06$ $τ = 1$ means ::: no target network (instant copy) Double DQN with online argmax $= 0$ , target $[2.2, 1.5, 2.8]$ , $r = 0.5, γ = 0.9$ ::: $0.5 + 0.9 (2.2) = 2.48$ Max of $[- 5, - 1, - 8]$ ::: $- 1$ (closest to zero)

Connections

Deep Q-Networks (DQN) — the base algorithm these numbers live inside.
Bellman Equation — the backup every target is a snapshot of.
Experience Replay — the other stabiliser; supplies the transitions we compute $y$ for.
Double DQN — Ex 8's decoupling trick.
DDPG / TD3 / SAC — soft-update (Ex 5) users.
Deadly Triad — Ex 7 is a miniature of the divergence it warns about.
Two-Timescale Stochastic Approximation — the theory behind Ex 6's slow $τ$ .