Double DQN and Dueling DQN

0. Prerequisite recap: what plain DQN does

WHY the target network? If we bootstrapped off $Q_\theta$ itself, the target moves every gradient step → we chase a moving goalpost → divergence. Freezing a copy $\theta^-$ makes the target approximately stationary for a few thousand steps.

1. Double DQN — curing the optimism

1.1 WHY does the $\max$ overestimate?

Formula Jensen-style overestimation bound (derived)

Let the true action values all equal $V$ , and let our estimates be $Q(a) = V + \varepsilon_a$ with $\varepsilon_a$ zero-mean, independent, variance $\sigma^2$ . Then the estimate we use in the target is $\max_a Q(a) = V + \max_a \varepsilon_a.$ Taking expectation: $\mathbb{E}\!\left[\max_a Q(a)\right] = V + \underbrace{\mathbb{E}\!\left[\max_a \varepsilon_a\right]}_{\;\geq\, 0}.$ Why $\geq 0$ ? For $m$ i.i.d. zero-mean variables, $\mathbb{E}[\max]\ge \mathbb{E}[\text{any single one}]=0$ , with equality only when $m=1$ . In fact for $m$ actions one can show $\mathbb{E}[\max_a\varepsilon_a] \le \sigma\frac{m-1}{\sqrt{2m-1}}$ — the bias grows with the number of actions $m$ .

HOW does this hurt learning? The inflated target propagates through the Bellman backup, is treated as ground truth, gets re-maxed next step... the error compounds and can make the agent prefer bad actions.

1.2 The fix: decouple selection from evaluation

Compare with plain DQN which uses $\theta^-$ for both: $y^{DQN} = r + \gamma\, Q_{\theta^-}\big(s',\ \arg\max_{a'} Q_{\theta^-}(s',a')\big).$ The change is one line of code, no new parameters.

Worked example Numeric demo of the bias

Suppose true $Q(s',\cdot)=0$ for 3 actions, but noisy estimates are:

Online $\theta$ : $[+0.3,\ -0.5,\ +0.1]$
Target $\theta^-$ : $[-0.2,\ +0.4,\ -0.1]$

Plain DQN: $\max_a Q_{\theta^-} = +0.4$ → target inflated to $r+\gamma(0.4)$ . Double DQN: selection: $\arg\max Q_\theta = a_1$ (value $+0.3$ ). Evaluate that same $a_1$ with $\theta^-$ : $Q_{\theta^-}(a_1)=-0.2$ → target $r+\gamma(-0.2)$ . Why this step? Because $a_1$ 's luck in $\theta$ is independent of $\theta^-$ 's reading, we don't lock in the optimistic value. Closer to the true $0$ .

2. Dueling DQN — a smarter network shape

2.1 WHY split value and advantage?

2.2 The architecture

The network splits into two streams after a shared feature trunk:

a scalar $V_\eta(s)$ (value stream),
a vector $A_\psi(s,a)$ (advantage stream),

then recombines them into $Q$ . Naïvely $Q=V+A$ , but:

Formula Dueling aggregation (mean-centered, the practical choice)

$\boxed{\,Q(s,a) = V(s) + \Big(A(s,a) - \tfrac{1}{|\mathcal A|}\textstyle\sum_{a'} A(s,a')\Big)\,}$ Derivation of why this fixes identifiability. Impose the constraint that advantages must sum to zero across actions, i.e. $\sum_{a'} A(s,a')=0$ (a valid choice since advantages are relative). Given any raw stream output $\tilde A$ , subtract its mean: $A(s,a) = \tilde A(s,a) - \frac{1}{|\mathcal A|}\sum_{a'}\tilde A(s,a').$ Now $\sum_a A(s,a)=0$ is guaranteed, so $V$ is pinned to the average $Q$ : summing the boxed equation over $a$ gives $\sum_a Q = |\mathcal A|\,V$ , i.e. $V(s)=\frac1{|\mathcal A|}\sum_a Q(s,a)$ . Unique split ✔.

An alternative uses $\max$ instead of mean: $Q = V + (A - \max_{a'}A)$ , which pins $V=\max_a Q = V^{best}$ . The mean version is more stable (subtracting the mean is a gentler, lower-variance operation than subtracting a moving argmax).

2.3 Why Dueling helps — the payoff

Dueling is orthogonal to Double — you combine both: dueling network architecture + double-DQN target. That combo ("Double Dueling DQN") is standard.

3. Putting it together (mnemonic + recap)

Recall Feynman: explain to a 12-year-old

Imagine picking the "best" ice-cream flavour by tasting one tiny spoon of each. Sometimes a bad flavour tastes great on that one lucky spoon, so you overrate it. Double DQN = one friend picks the flavour, a different friend tastes it to score it — so a lucky spoon doesn't fool you. Dueling DQN = instead of scoring each flavour from scratch, first ask "is this ice-cream shop even good?" (that's $V$ ), then only ask "is chocolate a bit better or worse than average here?" (that's $A$ ). You learn the shop's quality from any flavour you try, so you learn faster.

Connections

Deep Q-Networks (DQN) — the base algorithm both extend.
Target Networks and Experience Replay — $\theta^-$ reused for evaluation in Double DQN.
Advantage Function — the $A^\pi$ that Dueling factors out.
Overestimation Bias in Q-learning — general phenomenon; Double Q-learning (Hasselt 2010) is the tabular ancestor.
Actor-Critic Methods — also separate value ( $V$ ) from policy/advantage.
Rainbow DQN — combines Double + Dueling + Prioritized Replay + more.

Flashcards

Why does the max operator in DQN cause overestimation?

Estimates are noisy; taking the max preferentially selects actions with upward noise, so

\mathbb{E}[\max_a Q]\ge \max_a \mathbb{E}[Q]

— a positive bias that grows with the number of actions.

Write the Double DQN target.

y = r + \gamma\, Q_{\theta^-}(s', \arg\max_{a'} Q_{\theta}(s',a'))

— select the action with the online net

\theta

, evaluate it with the target net

\theta^-

What does plain DQN use to both select and evaluate the next action?

The target network

\theta^-

for both, which is exactly why the noise is double-counted.

How many new parameters does Double DQN add?

Zero — it only changes which network selects vs evaluates.

Define the advantage function.

A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)

: how much better action

a

is than the state's average.

What are the two streams in a Dueling network?

A scalar state-value stream

V(s)

and a per-action advantage stream

A(s,a)

, sharing a feature trunk.

Why can't we just set

Q = V + A

directly?

Non-identifiability: adding

c

V

and subtracting

c

from all

A

gives identical

Q

, so the split is unstable/undefined.

Give the mean-centered dueling aggregation.

Q(s,a)=V(s)+\big(A(s,a)-\frac1{|\mathcal A|}\sum_{a'}A(s,a')\big)

Under mean-centering, what does

V(s)

equal?

The mean of

Q

over actions:

V(s)=\frac1{|\mathcal A|}\sum_a Q(s,a)

Why is mean-centering preferred over max-centering in dueling?

The mean is a smoother, lower-variance anchor; the max shifts with a possibly-changing argmax, hurting stability.

Why does the dueling value stream learn faster?

V(s)

is updated by every transition regardless of action taken, so states get good values even for rarely-tried actions.

Can Double and Dueling be combined?

Yes — dueling is an architecture, double is a target rule; combined they give "Double Dueling DQN" (used in Rainbow).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, plain DQN mein ek chhoti si beemari hai: jab hum target banate waqt $\max_{a'} Q(s',a')$ lete hain, toh humara $Q$ estimate thoda noisy hota hai. Max lene se hum us action ko chun lete hain jiske paas is baar lucky noise zyada tha. Isse target humesha thoda over-estimate ho jaata hai — agent zyada optimistic ban jaata hai. Double DQN ka jugaad simple hai: action choose karo online network $\theta$ se, lekin us action ki value evaluate karo target network $\theta^-$ se. Dono ka noise alag-alag hota hai, toh lucky-noise ka double counting ruk jaata hai. Sirf ek line ka change, koi naya parameter nahi.

Dueling DQN doosri problem solve karta hai. Bahut saare states mein action barely matter karta hai (jaise seedhi road pe steering se koi farak nahi). Toh har action ke liye alag-alag full $Q$ seekhna waste hai. Isliye network ko do streams mein tod dete hain: ek batata hai "yeh state kitni achhi hai" = $V(s)$ , doosra batata hai "yeh action average se kitna behtar/kharab hai" = advantage $A(s,a)$ . Fayda: $V(s)$ har transition se update hota hai, chahe koi bhi action liya ho — isse learning fast ho jaati hai.

Ek catch hai: agar seedha $Q=V+A$ likh doge toh split unique nahi rehta ( $V$ mein $+c$ aur $A$ mein $-c$ karo, $Q$ same rahega). Isliye hum advantage ka mean subtract karte hain: $Q=V+(A-\overline{A})$ . Isse $V$ ban jaata hai average $Q$ , aur split fix ho jaata hai. Mean use karte hain (max nahi) kyunki mean stable aur smooth hai.

Yaad rakhna: Double = selection online, evaluation target; Dueling = V plus A minus mean. Dono orthogonal hain, saath mein use kar sakte ho — Rainbow DQN mein yahi combo hota hai. Interview aur real projects dono mein ye standard tricks hain.

0. Prerequisite recap: what plain DQN does

1. Double DQN — curing the optimism

1.1 WHY does the $\max$ overestimate?

1.2 The fix: decouple selection from evaluation

2. Dueling DQN — a smarter network shape

2.1 WHY split value and advantage?

2.2 The architecture

2.3 Why Dueling helps — the payoff

3. Putting it together (mnemonic + recap)

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Deep & Advanced RL

Connections