Target networks: Question bank — AI-ML deep dive

Before we begin, three plain-language reminders so every symbol is earned:

$θ$ (theta) = the online network's dials — the ones gradient descent turns every step.
$θ^{-}$ (theta-minus) = the target network's dials — a lagged, frozen-ish copy.
$y$ = the target label we regress toward, built from the Bellman idea $y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ , where $r$ is reward, $γ$ (gamma) is the discount factor between 0 and 1, and $max_{a^{'}}$ picks the best next-state action value.

True or false — justify

True or false: A target network changes what value Q-learning converges to.

False. It changes the stability of the path to the fixed point, not the fixed point itself — the Bellman optimum is the same; the target net just stops us oscillating on the way there.

True or false: With

τ = 1

the soft update is identical to having no target network at all.

True.

θ^{-} \leftarrow 1 \cdot θ + 0 \cdot θ^{-}

means

θ^{-}

becomes

θ

every step, so target and prediction share the same dials — exactly the "chasing your tail" setup.

True or false: Gradient descent should backpropagate through

θ^{-}

so the target learns too.

False. The whole point is that

y

is a frozen constant for the update; if gradient flowed through

θ^{-}

the goal-post would move with the shot again, reintroducing the feedback loop.

True or false: The target network needs its own separate loss function and optimiser.

False. It is never trained by gradient descent — it only receives copies (hard) or blends (soft) of

θ

. There is one loss, one optimiser, acting on

θ

alone.

True or false: A very large hard-update period

C

is always safer than a small one.

False. Larger

C

is more stable but slower — the target grows stale, so the online net regresses toward outdated values. Stability and learning speed trade off; extreme

C

hurts.

True or false: Soft updates and hard updates give mathematically the same target-tracking behaviour.

False in detail, similar in spirit. Soft update is a smooth exponential moving average (memory

\sim 1/ τ

); hard update is a step function that jumps then freezes. Both slow the target, but the drift profile differs.

True or false: Experience replay and target networks solve the same problem.

False. Replay breaks temporal correlation between consecutive samples; the target network stabilises the moving-target feedback. They are the two independent stabilisers of DQN.

True or false: If your problem has no bootstrapping (pure supervised regression), you still need a target network.

False. The instability comes from the target depending on

θ

via the max-Q bootstrap. Remove bootstrapping and the label is already a fixed constant — nothing to freeze.

Spot the error

Spot the error: "Target:

y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ)

— I copy

θ

into

θ^{-}

so I'll just write

θ

The label must use

θ^{-}

, not

θ

. Even if

θ^{-}

was recently copied from

θ

, between copies it must stay pinned; writing

θ

means gradient can flow through it and the target moves — defeating the purpose.

Spot the error: "For a terminal transition I set

y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})

Terminal states have no successor, so there is no future value:

y = r

only. Querying

Q (s^{'}, \cdot)

for a nonexistent next state injects a spurious bootstrapped value — a classic silent bug.

Spot the error: "Double DQN removes the target network because it already fixes the max."

Double DQN still uses the target network — it evaluates the chosen action with

θ^{-}

. It only changes which value the max selects (online net selects, target net evaluates), addressing overestimation, not stability.

Spot the error: "Soft update:

θ^{-} \leftarrow τ θ^{-} + (1 - τ) θ

with

τ = 0.005

The weights are swapped. It should be

θ^{-} \leftarrow τ θ + (1 - τ) θ^{-}

: the small weight

τ

goes on the new online value, the large weight

1 - τ

on the old target. As written,

θ^{-}

would jump almost fully to

θ

each step.

Spot the error: "I take

ar g max

and

max

over

Q (s, a; θ^{-})

but the current state

s

, not

s^{'}

The bootstrap looks at the next state

s^{'}

, not the current

s

. The target is "reward now plus discounted best value of where we land," so it must evaluate the successor

s^{'}

Spot the error: "Since

θ^{-}

never gets a gradient, I can initialise it randomly and leave it — it doesn't matter."

It matters.

θ^{-}

must be initialised as a copy of

θ

and kept close to it via updates; a permanently random

θ^{-}

produces garbage labels and the online net regresses to noise.

Why questions

Why does using the same

θ

for target and prediction risk divergence, not just slowness?

Because reducing the error can move the target further, so the fixed-point iteration is not guaranteed to be a contraction — errors can amplify rather than shrink, especially with off-policy sampling and function approximation (the deadly triad).

Why is the soft update called an exponential moving average?

Unrolling

θ^{-} \leftarrow τ θ + (1 - τ) θ^{-}

shows the weight on a value

k

steps old decays as

(1 - τ)^{k}

— an exponential decay — so

θ^{-}

is a weighted average that forgets the past geometrically.

Why does two-timescale stochastic-approximation theory require the learning-rate ratio to go to zero?

So the fast online net effectively sees an (almost) stationary target: from the fast timescale's view the slow target looks frozen, turning the coupled problem into a solvable regression at each moment, which is what guarantees convergence.

Why does the max operator make the target specifically the frozen network's max, not the online one?

To keep the label constant during the fit. If the max were taken over the online net that we are simultaneously moving, the greedy action and its value could flip mid-update, and the label would shift — the exact instability we are removing.

Why can a target network mask rather than cure the deadly triad?

It tames the bootstrapping-instability leg by slowing the target, but off-policy sampling and function-approximation error remain. It buys stability and time; it does not make divergence impossible in all settings.

Why is small

τ

(say

0.005

) preferred over

τ = 0.5

in DDPG/TD3/SAC?

Continuous-control actor-critic methods are especially sensitive to target drift; a tiny

τ

gives an effective memory of

\sim 200

steps, keeping the critic's target smooth so the actor doesn't chase a jittering value estimate.

Edge cases

Edge case: What is the target when reward is

r = 0

and

s^{'}

is terminal?

y = 0

. Terminal kills the bootstrap term, and zero reward leaves nothing — a perfectly valid zero label, not an error to "fix."

Edge case: Right after a hard update

θ^{-} \leftarrow θ

, are target and prediction momentarily identical?

Yes, for that single step

θ^{-} = θ

, so briefly you are back in the coupled regime — but only for one step, before

θ

moves on and

θ^{-}

freezes again. This transient is harmless.

Edge case: In an environment where

γ = 0

, does the target network still do anything?

No. With

γ = 0

the target is

y = r

, a fixed reward — no bootstrap, no dependence on any network, so there is nothing to stabilise.

Edge case: If

τ

is so tiny that

θ^{-}

barely moves over the whole run, what fails?

The target becomes stale and never reflects the improving online net, so the online net regresses toward badly outdated values and learning stalls — the freeze-forever (

τ \to 0

) failure mode.

Edge case: Can two different states

s^{'}

share the same max action index in

Q (s^{'}, \cdot; θ^{-})

yet give different targets?

Yes. The max is taken per state; different

s^{'}

have different value vectors, so even the same greedy action index yields different

max Q

values and hence different

y

Connections

Deep Q-Networks (DQN) — where hard-update target nets originated.
Experience Replay — the orthogonal stabiliser.
Bellman Equation — supplies the bootstrapped label being frozen.
Double DQN — orthogonal fix for overestimation, uses the target net.
DDPG / TD3 / SAC — soft (Polyak) updates in continuous control.
Deadly Triad — the instability target nets tame.
Two-Timescale Stochastic Approximation — the convergence theory behind slow updates.