Exercises — Target networks

What we want: just recall the formula. $y=r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }^{-})$ The weights inside are ${\theta }^{-}$ — the target (frozen) network, not the online $\theta$. That is the whole point: the label $y$ must not shift when we nudge $\theta$.

False. We treat ${\theta }^{-}$ as a constant. The gradient is ${\nabla }_{\theta }L=-2(y-Q(s,a;\theta )){\nabla }_{\theta }Q(s,a;\theta ),$ and $y$ (which contains ${\theta }^{-}$) is held fixed — no derivative is taken w.r.t. ${\theta }^{-}$.

• Hard update: ${\theta }^{-}\leftarrow \theta$ every $C$ steps.
• Soft update (Polyak): ${\theta }^{-}\leftarrow \tau \theta +(1-\tau ){\theta }^{-}$ with $0<\tau \ll 1$.

Step 1 — pick the max (WHY): the Bellman backup uses the best future action value, so we take ${\max }_{a^{\prime }}Q=4.0$. We use the frozen net so the label won't drift while we fit. Step 2 — discount and add reward: $y=r+\gamma {\max }_{a^{\prime }}Q=2+0.95\times 4.0=2+3.8=\mathrm{5.8.}$

TD error $=y-Q(s,a;\theta )=5.8-3.0=2.8>0$. Since the error is positive, the target is above the prediction, so gradient descent increases $Q(s,a;\theta )$ toward $5.8$. Crucially $y=5.8$ does not change this step (it depends only on ${\theta }^{-}$).

${\theta }_{\text{new}}^{-}=\tau \theta +(1-\tau ){\theta }^{-}=0.02(8)+0.98(3)=0.16+2.94=\mathrm{3.10.}$ It creeps only $0.10$ toward $8$ — the target moves $\approx 50\times$ slower ($1/\tau =50$).

WHY: a terminal state has no successor, so the Bellman backup drops the future term entirely. The frozen net is not even queried. $y=r=2.$

Step 1 — unroll (WHAT & WHY): starting from ${\theta }_0^{-}$ and holding $\theta$ fixed, ${\theta }_1^{-}=\tau \theta +(1-\tau ){\theta }_0^{-}.$ Applying the same rule again: ${\theta }_2^{-}=\tau \theta +(1-\tau ){\theta }_1^{-}=\tau \theta +(1-\tau )[\tau \theta +(1-\tau ){\theta }_0^{-}].$ By induction the coefficient on ${\theta }_0^{-}$ is $(1-\tau {)}^k$ — an exponential moving average. This is why we call the soft update an EMA. Step 2 — half-life: solve $(1-\tau {)}^k=\frac{1}{2}$ with $\tau =0.005$: $k=\frac{\ln (1/2)}{\ln (1-0.005)}=\frac{-0.6931}{-0.005013}\approx \mathrm{138.3.}$ So the target "half-forgets" its old value after about $138$ steps. Effective memory $\sim 1/\tau =200$ steps — same order of magnitude.

The picture: the black dashed line is the ideal fixed target; the red curve is $y(\theta )$ which also rises as $\theta$ rises because it shares $\theta$. When you push the prediction $\hat{q}$ up to close the gap, the target $y$ slides up too — you are chasing a goal-post you yourself pushed forward. The vertical red gap need not shrink. Formally: the coupled update is ${\theta }_{t+1}={\theta }_t+\alpha (y({\theta }_t)-\hat{q}({\theta }_t))\nabla \hat{q}.$ Because $\partial y/\partial \theta \ne 0$, the map need not be a contraction (a step need not bring you closer to the fixed point). Freezing $y$ at ${\theta }^{-}$ makes $\partial y/\partial \theta =0$ locally → ordinary supervised regression → stable.

Yes for the network weights: ${\theta }^{-}$ is literally unchanged for $C$ steps, so the function $Q(\cdot ,\cdot ;{\theta }^{-})$ is fixed. But $y$ can still differ across updates because $y$ also depends on the transition $(s^{\prime },r,\text{done})$ sampled from Experience Replay. Different mini-batch samples → different $s^{\prime }$ → different ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }^{-})$. Stability means the labeling function is frozen, not that every label is the same number.

Double DQN target (WHY): the online net selects the action, the target net evaluates it — this decouples selection from evaluation to cut over-estimation: $y=r+\gamma Q(s^{\prime },\arg {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };\theta );{\theta }^{-}).$ Step 1 — select with online: $\arg \max$ of $[2.0,7.0,3.0]$ is action index $2$ (the $7.0$). Step 2 — evaluate with target: target net's value at that action is $4.0$. $y=1+0.9\times 4.0=1+3.6=\mathrm{4.6.}$ Compare plain: plain target net uses $\max$ of the target outputs $=6.0$, giving $y_{\text{plain}}=1+0.9\times 6.0=\mathrm{6.4.}$ Double DQN's $4.6<6.4$: it refuses to reward the target net's own optimistic pick, reducing overestimation. See Double DQN.

Apply componentwise ${\theta }_{\text{new}}^{-}=\tau \theta +(1-\tau ){\theta }^{-}$:

• Component 1: $0.05(10)+0.95(2)=0.5+1.9=2.4$.
• Component 2: $0.05(-4)+0.95(-1)=-0.2-0.95=-1.15$. ${\theta }_{\text{new}}^{-}=[2.4,-1.15].$ These continuous-control algorithms (DDPG, TD3, SAC) prefer this smooth creep over abrupt copies.

Condition: the target must move on a slower timescale, i.e. $\frac{\tau }{\alpha }\to 0$ (target rate $\ll$ online rate). Then Two-Timescale Stochastic Approximation guarantees the fast online net effectively sees a quasi-static target and converges.

• $\tau \to 1$: ${\theta }^{-}$ = $\theta$ always → no target network → we are back to chasing our own tail (the Deadly Triad bites: bootstrapping + off-policy + approximation → oscillation/divergence).
• $\tau \to 0$: ${\theta }^{-}$ is frozen forever → the online net regresses to a target that never improves → learning stalls at a stale value.

Step 1 — effective memory: memory $\approx 1/\tau$. Current $=1/0.02=50$ steps. Tripling → $150$ steps → set $\tau =1/150\approx 0.006667$. Step 2 — half-life at new $\tau$: $k=\frac{\ln (1/2)}{\ln (1-\tau )}=\frac{-0.6931}{\ln (0.993333)}=\frac{-0.6931}{-0.006689}\approx 103.6\text{ steps}.$ So the new target half-forgets after $\approx 104$ steps, roughly $3\times$ slower than before (old half-life $\approx 34.3$ steps).

The picture: the red staircase is the hard update (jumps of full size every $C$ steps); the black smooth curve is the soft EMA (tiny steps every step). Both reach the online value over a comparable horizon. Argument: hold $\theta$ constant at a new value ${\theta }^{\ast }$; the old target value is ${\theta }_0^{-}$. Under soft updates the gap to ${\theta }^{\ast }$ after $k$ steps is ${\theta }^{\ast }-{\theta }_k^{-}=(1-\tau {)}^k({\theta }^{\ast }-{\theta }_0^{-}).$ Setting $k=C=1/\tau$ and using $(1-\tau {)}^{1/\tau }\to e^{-1}\approx 0.368$ for small $\tau$: after $C$ soft steps roughly $63\%$ of the gap is closed — the target has "mostly caught up" over exactly the horizon $C=1/\tau$, which is precisely when a hard update would have copied it fully. Hence the timescales match at $C\approx 1/\tau$.