Double DQN and Dueling DQN — Hinglish notes

5.2.4 · AI-ML › Deep & Advanced RL

0. Prerequisite recap: plain DQN kya karta hai

Target network kyun? Agar hum $Q_{θ}$ se hi bootstrap karte, toh target har gradient step par hilta → hum ek hilte hue goalpost ka peecha karte → divergence. Ek copy $θ^{-}$ ko freeze karna target ko kuch hazaar steps ke liye approximately stationary banata hai.

1. Double DQN — optimism ka ilaaj

1.1 $max$ overestimate kyun karta hai?

Yeh learning ko kaise nuksan pahunchata hai? Inflated target Bellman backup mein propagate hota hai, ground truth maana jaata hai, agli step par phir se max hota hai... error compound hota jaata hai aur agent ko bure actions prefer karaata hai.

1.2 Fix: selection aur evaluation ko alag karo

Plain DQN se compare karo jo dono ke liye $θ^{-}$ use karta hai: $y^{D QN} = r + γ Q_{θ^{-}} (s^{'}, ar g max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'})) .$ Change code ki ek line hai, koi naye parameters nahi.

2. Dueling DQN — ek smarter network shape

2.1 Value aur advantage ko split kyun karein?

2.2 Architecture

Network ek shared feature trunk ke baad do streams mein split hota hai:

ek scalar $V_{η} (s)$ (value stream),
ek vector $A_{ψ} (s, a)$ (advantage stream),

phir inhe $Q$ mein recombine karta hai. Naively $Q = V + A$ , lekin:

Formula Dueling aggregation (mean-centered, practical choice)

$Q (s, a) = V (s) + (A (s, a) - \frac{1}{∣ A ∣} \sum_{a^{'}} A (s, a^{'}))$ Identifiability fix kaise hoti hai, iska derivation. Constraint lagao ki advantages ko actions mein sum zero hona chahiye, yani $\sum_{a^{'}} A (s, a^{'}) = 0$ (valid choice kyunki advantages relative hain). Kisi bhi raw stream output $\tilde{A}$ ko diya gaya, uska mean ghataao: $A (s, a) = \tilde{A} (s, a) - \frac{1}{∣ A ∣} \sum_{a^{'}} \tilde{A} (s, a^{'}) .$ Ab $\sum_{a} A (s, a) = 0$ guaranteed hai, toh $V$ average $Q$ par pin hota hai: boxed equation ko $a$ par sum karo toh $\sum_{a} Q = ∣ A ∣ V$ milta hai, yani $V (s) = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)$ . Unique split ✔.

Ek alternative mean ke bajaye $max$ use karta hai: $Q = V + (A - max_{a^{'}} A)$ , jo $V = max_{a} Q = V^{b es t}$ pin karta hai. Mean version zyada stable hai (mean subtract karna moving argmax subtract karne se gentler, lower-variance operation hai).

2.3 Dueling kyun help karta hai — payoff

Dueling, Double se orthogonal hai — tum dono combine kar sakte ho: dueling network architecture + double-DQN target. Yeh combo ("Double Dueling DQN") standard hai.

3. Sab kuch ek saath (mnemonic + recap)

Recall Feynman: ek 12-saal ke bachche ko samjhaao

Socho tum "best" ice-cream flavour choose kar rahe ho ek ek tiny spoon taste karke. Kabhi kabhi ek buri flavour us ek lucky spoon par great lagti hai, toh tum usse overrate kar dete ho. Double DQN = ek dost flavour choose karta hai, ek alag dost use score karne ke liye taste karta hai — toh ek lucky spoon tumhe fool nahi kar sakta. Dueling DQN = har flavour ko scratch se score karne ke bajaye, pehle poochho "kya yeh ice-cream shop achhi hai?" (woh hai $V$ ), phir sirf poochho "kya chocolate yahan average se thoda better ya worse hai?" (woh hai $A$ ). Tum shop ki quality kisi bhi flavour se try karke seekhte ho, toh tum tezi se seekhte ho.

Connections

Deep Q-Networks (DQN) — base algorithm jise dono extend karte hain.
Target Networks and Experience Replay — Double DQN mein evaluation ke liye $θ^{-}$ reuse hota hai.
Advantage Function — woh $A^{π}$ jo Dueling factor out karta hai.
Overestimation Bias in Q-learning — general phenomenon; Double Q-learning (Hasselt 2010) tabular ancestor hai.
Actor-Critic Methods — value ( $V$ ) ko policy/advantage se alag karte hain.
Rainbow DQN — Double + Dueling + Prioritized Replay + aur combine karta hai.

Flashcards

DQN mein max operator overestimation kyun karta hai?

Estimates noisy hain; max lena preferentially un actions ko select karta hai jinki upward noise hai, toh

E [max_{a} Q] \geq max_{a} E [Q]

— ek positive bias jo actions ki sankhya ke saath badhta hai.

Double DQN target likhho.

y = r + γ Q_{θ^{-}} (s^{'}, ar g max_{a^{'}} Q_{θ} (s^{'}, a^{'}))

— online net

θ

se action select karo, target net

θ^{-}

se evaluate karo.

Plain DQN agli action ko select aur evaluate karne ke liye kya use karta hai?

Dono ke liye target network

θ^{-}

, yehi wajah hai ki noise double-count hoti hai.

Double DQN kitne naye parameters add karta hai?

Zero — sirf yeh change karta hai ki kaunsa network select karta hai vs evaluate karta hai.

Advantage function define karo.

A^{π} (s, a) = Q^{π} (s, a) - V^{π} (s)

: action

a

state ke average se kitna better hai.

Dueling network ke do streams kya hain?

Ek scalar state-value stream

V (s)

aur ek per-action advantage stream

A (s, a)

, shared feature trunk ke saath.

Hum seedha

Q = V + A

kyun nahi set kar sakte?

Non-identifiability:

V

mein

c

add karo aur sab

A

mein se

c

ghataao toh identical

Q

milta hai, toh split unstable/undefined hai.

Mean-centered dueling aggregation do.

Q (s, a) = V (s) + (A (s, a) - \frac{1}{∣ A ∣} \sum_{a^{'}} A (s, a^{'}))

Mean-centering ke under,

V (s)

kya equal hai?

Actions par

Q

ka mean:

V (s) = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)

Dueling mein mean-centering ko max-centering se zyada kyun prefer kiya jaata hai?

Mean ek smoother, lower-variance anchor hai; max possibly-changing argmax ke saath shift karta hai, stability hurt karta hai.

Dueling value stream tezi se kyun seekhta hai?

V (s)

har transition par update hota hai chahe koi bhi action liya ho, toh states ko achhe values milte hain chahe actions rarely try ki gayi hon.

Kya Double aur Dueling combine ho sakte hain?

Haan — dueling ek architecture hai, double ek target rule hai; combine hoke yeh "Double Dueling DQN" dete hain (Rainbow mein use hota hai).