Target networks — Hinglish notes

5.2.3 · AI-ML › Deep & Advanced RL

PROBLEM kya hai?

Q-learning with function approximation (jaise DQN) mein, hum ek network $Q (s, a; θ)$ ko train karte hain taaki woh Bellman equation satisfy kare. Training target hota hai:

$y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ)$

Phir hum temporal-difference (TD) loss minimise karte hain:

$L (θ) = (target y r + γ a^{'} max Q (s^{'}, a^{'}; θ) - prediction Q (s, a; θ))^{2}$

HOW: ek doosra parameter set introduce karo

Do networks rakho:

Online (learner) network $Q (s, a; θ)$ — har step mein gradient descent se update hota hai.
Target network $Q (s, a; θ^{-})$ — ek lagged copy jo sirf targets compute karne ke liye use hoti hai.

Target ban jaata hai:

$y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; \overset{=}{ˉ} θ^{-} ==)$

aur loss:

$L (θ) = (r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-}) - Q (s, a; θ))^{2}$

Sabse important baat, hum $θ^{-}$ ko constant treat karte hain — iske through koi gradient flow nahi hota. Toh gradient hai:

$\nabla_{θ} L = - 2 (y - Q (s, a; θ)) \nabla_{θ} Q (s, a; θ)$

Target $y$ ab kaafi steps ke liye ek stationary supervised label hai → problem locally standard regression jaisi lagti hai, jo stable hoti hai.

$θ^{-}$ update karne ke do tarike

1. Hard update (periodic copy) — original DQN mein use hua: $θ^{-} \leftarrow θ every C steps$ Copies ke beech mein, $θ^{-}$ frozen rahta hai.

2. Soft update (Polyak averaging) — DDPG / TD3 / SAC mein use hota hai: $θ^{-} \leftarrow τ θ + (1 - τ) θ^{-}, 0 < τ ≪ 1$

Timescale derive karna: lag kyun help karta hai

Target ko ek moving point $y (θ)$ socho aur prediction $\overset{q}{^} (θ) = Q (s, a; θ)$ .

Coupled update (koi target net nahi): $θ_{t + 1} = θ_{t} + α (y (θ_{t}) - \overset{q}{^} (θ_{t})) \nabla \overset{q}{^}$

Kyunki $y$ bhi tab increase hota hai jab $\overset{q}{^}$ hota hai (dono share karte hain $θ$ ), "error" $y - \overset{q}{^}$ shrink hone mein fail ho sakta hai — fixed-point iteration contraction hone ki guarantee nahi hai. $y$ ko $θ^{-}$ par pin karke jo slower timescale par change hota hai, hum ek two-timescale system paate hain: fast $θ$ ek slowly-drifting target ki taraf regression solve karta hai. Stochastic-approximation theory kehti hai aise two-timescale schemes converge karte hain jab learning rates ka ratio $\to 0$ ho. Yahi formal justification hai $τ ≪ 1$ ya bade $C$ ke liye.

Worked examples

Flashcards

DQN mein target network kyun chahiye?

TD target

r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ)

depend karta hai

θ

par;

θ

update karna us target ko move karta hai jise aap chase kar rahe ho, correlated feedback → oscillation/divergence create karta hai. Ek frozen copy stable target deti hai.

Target network update karne ke do tarike kya hain?

Hard update:

θ^{-} \leftarrow θ

har

C

steps par copy karo. Soft update:

θ^{-} \leftarrow τ θ + (1 - τ) θ^{-}

chote

τ

ke saath.

Soft update mein

τ

kya control karta hai?

Tracking speed / effective memory (

\sim 1/ τ

steps).

τ \to 1

= koi target net nahi;

τ \to 0

= hamesha ke liye frozen.

Kya gradient

θ^{-}

ke through flow karta hai?

Nahi —

θ^{-}

ko constant treat kiya jaata hai; target ek stationary label hai.

Terminal transition ke liye TD target kya hota hai?

Sirf

y = r

(koi bootstrapped

γ max Q

term nahi).

Double DQN target network se alag kaise hai?

Double DQN max-overestimation bias reduce karta hai online net se action select karke aur target net se evaluate karke; yeh target network ke stability role se orthogonal hai.

Slow target updates ko kaunsi theory justify karti hai?

Two-timescale stochastic approximation: fast online net slowly-drifting target ki taraf regress karta hai; convergence ke liye learning-rate ratio → 0 chahiye.

Target network ke saath full TD loss likhiye.

L (θ) = (r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-}) - Q (s, a; θ))^{2}

Recall Feynman: 12-saal ke bacche ko explain karo

Socho tum archery practice kar rahe ho, lekin target har baar jab tum shoot karte ho move karta hai — aur woh tumhare apne shots ki wajah se move karta hai. Tum kabhi kuch nahi maar paoge! Toh instead, hum target ko thodi der ke liye freeze kar dete hain (ya ise bahut slowly drift hone dete hain). Tum frozen target ko hit karne ki practice karte ho, acche ho jaate ho, phir target ko thoda sa move karte ho, phir practice karte ho. Woh "frozen bullseye" hi target network hai — yeh learning ko chaotic ki jagah calm aur steady banata hai.

Connections

Deep Q-Networks (DQN) — target networks DQN ke do key stabilisers mein se ek hain.
Experience Replay — DQN ka doosra stabiliser (temporal correlations tod deta hai).
Bellman Equation — bootstrapped target supply karta hai jo freeze hota hai.
Double DQN — target nets ke saath combine hota hai overestimation cut karne ke liye.
DDPG / TD3 / SAC — soft (Polyak) target updates use karte hain.
Deadly Triad — bootstrapping + off-policy + approximation; target nets ise tame karte hain.
Two-Timescale Stochastic Approximation — slow updates ke liye theoretical backing.