Noesis
5.2.3Deep & Advanced RL

Target networks

1,670 words8 min readdifficulty · medium2 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is the problem?

In Q-learning with function approximation (e.g. DQN), we train a network Q(s,a;θ)Q(s,a;\theta) to satisfy the Bellman equation. The training target is:

y=r+γmax⁡a′Q(s′,a′;θ)y = r + \gamma \max_{a'} Q(s', a'; \theta)

We then minimise the temporal-difference (TD) loss:

L(θ)=(  r+γmax⁡a′Q(s′,a′;θ)⏟target y−Q(s,a;θ)⏟prediction  )2L(\theta) = \Big(\; \underbrace{r + \gamma \max_{a'} Q(s',a';\theta)}_{\text{target } y} - \underbrace{Q(s,a;\theta)}_{\text{prediction}} \;\Big)^2

HOW: introduce a second parameter set

Keep two networks:

  • Online (learner) network Q(s,a;θ)Q(s,a;\theta) — updated every step by gradient descent.
  • Target network Q(s,a;θ−)Q(s,a;\theta^-) — a lagged copy used only to compute targets.

The target becomes:

y=r+γmax⁡a′Q(s′,a′;=ˉθ−==)y = r + \gamma \max_{a'} Q(s',a';\==\theta^-==)

and the loss:

L(θ)=( r+γmax⁡a′Q(s′,a′;θ−)−Q(s,a;θ) )2L(\theta) = \big(\,r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta)\,\big)^2

Crucially, we treat θ−\theta^- as a constant — no gradient flows through it. So the gradient is:

∇θL=−2(y−Q(s,a;θ)) ∇θQ(s,a;θ)\nabla_\theta L = -2\big(y - Q(s,a;\theta)\big)\,\nabla_\theta Q(s,a;\theta)

The target yy is now a stationary supervised label for many steps → the problem locally looks like standard regression, which is stable.

Two ways to update θ−\theta^-

1. Hard update (periodic copy) — used in the original DQN: θ−←θevery C steps\theta^- \leftarrow \theta \quad \text{every } C \text{ steps} Between copies, θ−\theta^- is frozen.

2. Soft update (Polyak averaging) — used in DDPG / TD3 / SAC: θ−←τ θ+(1−τ) θ−,0<τ≪1\theta^- \leftarrow \tau\,\theta + (1-\tau)\,\theta^-, \qquad 0 < \tau \ll 1

Figure — Target networks

Deriving the timescale: why lag helps

Think of the target as a moving point y(θ)y(\theta) and the prediction q^(θ)=Q(s,a;θ)\hat q(\theta)=Q(s,a;\theta).

Coupled update (no target net): θt+1=θt+α(y(θt)−q^(θt))∇q^\theta_{t+1} = \theta_t + \alpha\big(y(\theta_t) - \hat q(\theta_t)\big)\nabla \hat q

Because yy also increases when q^\hat q does (both share θ\theta), the "error" y−q^y-\hat q can fail to shrink — the fixed-point iteration is not guaranteed to be a contraction. By pinning yy to θ−\theta^- that changes on a slower timescale, we get a two-timescale system: fast θ\theta solves a regression toward a slowly-drifting target. Stochastic-approximation theory says such two-timescale schemes converge when the ratio of learning rates →0\to 0. That is the formal justification for τ≪1\tau\ll1 or large CC.

Worked examples

Flashcards

Why do we need a target network in DQN?
The TD target r+γmax⁡a′Q(s′,a′;θ)r+\gamma\max_{a'}Q(s',a';\theta) depends on θ\theta; updating θ\theta moves the target you're chasing, causing correlated feedback → oscillation/divergence. A frozen copy gives a stable target.
What are the two ways to update the target network?
Hard update: copy θ−←θ\theta^-\leftarrow\theta every CC steps. Soft update: θ−←τθ+(1−τ)θ−\theta^-\leftarrow\tau\theta+(1-\tau)\theta^- with small τ\tau.
In the soft update, what does τ\tau control?
The tracking speed / effective memory (∼1/τ\sim 1/\tau steps). τ→1\tau\to1 = no target net; τ→0\tau\to0 = frozen forever.
Does gradient flow through θ−\theta^-?
No — θ−\theta^- is treated as a constant; the target is a stationary label.
What is the TD target for a terminal transition?
y=ry=r only (no bootstrapped γmax⁡Q\gamma\max Q term).
How does Double DQN differ from a target network?
Double DQN reduces max-overestimation bias by selecting the action with the online net and evaluating it with the target net; it's orthogonal to the stability role of the target network.
What theory justifies slow target updates?
Two-timescale stochastic approximation: fast online net regresses to a slowly-drifting target; convergence needs the learning-rate ratio → 0.
Write the full TD loss with a target network.
L(θ)=(r+γmax⁡a′Q(s′,a′;θ−)−Q(s,a;θ))2L(\theta)=\big(r+\gamma\max_{a'}Q(s',a';\theta^-)-Q(s,a;\theta)\big)^2.
Recall Feynman: explain to a 12-year-old

Imagine practising archery, but the target keeps moving every time you shoot — and it moves because of your own shots. You'd never hit anything! So instead, we freeze the target for a while (or let it drift very slowly). You practise hitting the frozen target, get good, then move the target a tiny bit, practise again. That "frozen bullseye" is the target network — it makes learning calm and steady instead of chaotic.

Connections

  • Deep Q-Networks (DQN) — target networks are one of DQN's two key stabilisers.
  • Experience Replay — the other DQN stabiliser (breaks temporal correlations).
  • Bellman Equation — supplies the bootstrapped target being frozen.
  • Double DQN — combines with target nets to cut overestimation.
  • DDPG / TD3 / SAC — use soft (Polyak) target updates.
  • Deadly Triad — bootstrapping + off-policy + approximation; target nets tame it.
  • Two-Timescale Stochastic Approximation — theoretical backing for slow updates.

Concept Map

plugged into

target depends on theta

amplified by

leads to

computes

frozen params give

prevents

gradient descent on

copied into

updates every C steps

tracks smoothly

memory ~ 1 over tau

Bellman target y

TD loss squared error

Shared theta for target and prediction

Moving goal-post feedback loop

Deadly triad

Oscillation and divergence

Online network theta

Target network theta-minus

Stationary supervised label

Hard update periodic copy

Soft update Polyak EMA

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, DQN mein hum ek neural network se Q(s,a)Q(s,a) predict karte hain, aur uska target hota hai r+γmax⁡Q(s′,a′)r + \gamma \max Q(s',a'). Problem yeh hai ki yeh target bhi usi network se aata hai. Matlab jab aap weights update karte ho prediction sudharne ke liye, to target khud hi hil jaata hai — jaise aap teer maar rahe ho par nishaana har shot pe khisak raha hai. Isse training oscillate karti hai ya diverge ho jaati hai (deadly triad ka chakkar).

Solution simple hai: ek target network rakho — yeh online network ki ek purani, frozen copy hai (θ−\theta^-). Target (yy) hamesha is frozen copy se compute karo, aur is copy ke through koi gradient mat bahne do. Ab target thodi der ke liye ek fixed label ban jaata hai, aur problem normal supervised regression jaisi stable ho jaati hai.

Copy update karne ke do tareeke hain. Hard update: har CC steps pe θ−←θ\theta^- \leftarrow \theta (poora copy). Soft update (Polyak): har step θ−←τθ+(1−τ)θ−\theta^- \leftarrow \tau\theta + (1-\tau)\theta^- with chhota τ\tau jaise 0.0050.005 — yeh target ko dheere-dheere online net ke peeche khinchta hai. Yaad rakho: target net stability ke liye hai, aur Double DQN alag cheez hai jo overestimation bias kam karta hai — dono saath use hote hain, confuse mat karna.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,123 wordsD2Visual walkthroughThe derivation in pictures · 2,248 wordsD3Worked examplesEvery scenario covered · 2,918 wordsD4ExercisesGraded, full solutions · 2,257 wordsD5Question bankConcept traps, quick reveals · 1,518 words

Test yourself — Deep & Advanced RL

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Deep Q-Networks (DQN)AI-ML · 5.2.1Experience replayAI-ML · 5.2.2Double DQN and Dueling DQNAI-ML · 5.2.4Experience replayAI-ML · 5.2.2Double DQN and Dueling DQNAI-ML · 5.2.4