Double DQN and Dueling DQN: Question bank — AI-ML deep dive

This page hunts the misconceptions the parent topic invites. Each item is a one-line reveal: read the prompt, answer out loud, then check. Answers give the reasoning, never a bare yes/no.

Prerequisites worth having fresh: Deep Q-Networks (DQN), Target Networks and Experience Replay, Overestimation Bias in Q-learning, Advantage Function.

True or false — justify

TF1. Double DQN adds new trainable parameters to fix overestimation.

False. It reuses the existing online net

θ

and target net

θ^{-}

; the only change is who selects vs who evaluates the action — one line of code, zero new weights.

TF2. If the

Q

estimates were perfectly noise-free, plain DQN's

max

would still overestimate.

False. Overestimation comes from

E [max] \geq max E

on noisy values; with zero noise

max_{a} Q

equals the true

max

and there is no upward bias.

TF3. The overestimation bias is independent of how many actions the agent has.

False. For

m

i.i.d. zero-mean errors the bias grows with

m

(bounded by

σ \frac{m - 1}{2 m - 1}

); more actions means more chances for one to catch lucky noise.

TF4. In Dueling DQN,

Q (s, a) = V (s) + A (s, a)

(raw, uncentered) is a valid recombination.

False. It is not identifiable: adding

c

V

and subtracting

c

from every

A

leaves

Q

unchanged, so the two streams drift and

V

loses meaning. You must anchor the split (subtract the mean advantage).

TF5. Dueling DQN and Double DQN are competing methods; you pick one.

False. They are orthogonal — Dueling changes network shape, Double changes the target formula. "Double Dueling DQN" combines both and is standard.

TF6. With mean-centering,

V (s)

equals the best action's

Q

-value.

False. Mean-centering pins

V (s) = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)

, the average

Q

. It is the max-centering variant that pins

V

to the best action's value.

TF7. Double DQN completely eliminates overestimation bias.

False. It reduces it by decoupling selection from evaluation, but

θ

and

θ^{-}

are correlated (one is a lagged copy of the other), so their noise isn't fully independent — residual bias remains.

TF8. The target network exists specifically to solve the overestimation problem.

False. The target network's job is stability (a near-stationary bootstrap target). Overestimation persists even with a target net; Double DQN is a separate fix layered on top.

TF9. Because advantages are forced to sum to zero, at least one action always has a non-positive advantage.

True. If

\sum_{a} A (s, a) = 0

and they aren't all zero, some must be

\leq 0

; the below-average actions carry the negative advantages that balance the above-average ones.

TF10. Dueling DQN helps most when the choice of action strongly changes the outcome in every state.

False. It helps most when many states are low-stakes (all actions similar) — there the shared

V

stream learns the state's worth from every transition regardless of action.

Spot the error

SE1. "Double DQN selects the action with the target net and evaluates it with the online net."

Reversed. Correct is select with online $θ$ , evaluate with target $θ^{-}$ ("SO-ET"). The mnemonic direction matters — swapping them doesn't decouple the same way the derivation intends.

SE2. "Plain DQN target:

r + γ Q_{θ^{-}} (s^{'}, ar g max_{a^{'}} Q_{θ} (s^{'}, a^{'}))

That is the Double DQN target. Plain DQN uses

θ^{-}

for both select and evaluate:

r + γ Q_{θ^{-}} (s^{'}, ar g max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'})) = r + γ max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'})

SE3. "

E [max_{a} ε_{a}] \leq 0

because the errors are zero-mean."

Wrong sign.

E [max] \geq E [any single one] = 0

, so the bias is

\geq 0

. Zero-mean per variable does not make the max of them zero-mean.

SE4. "Given

V = 5

and raw

\tilde{A} = [3, 1, 2]

, dueling outputs

Q = [8, 6, 7]

Missing the centering. Mean of

\tilde{A}

2

; centered

A = [1, - 1, 0]

Q = 5 + [1, - 1, 0] = [6, 4, 5]

. Skipping centering lets the

+ 2

offset leak into

Q

and breaks

V = mean (Q)

SE5. "The mean-centered advantages recover the true advantage function exactly."

No. Centering only guarantees a unique, identifiable split; the learned

V, A

need not equal the true

V^{π}, A^{π}

. They are internal decompositions that reproduce the correct

Q

SE6. "Since Double DQN changes the target, it needs a separate target network from plain DQN."

No new network. Double DQN uses the same online/target pair DQN already has — it just routes selection to

θ

and evaluation to

θ^{-}

SE7. "

A^{π} (s, a) = V^{π} (s) - Q^{π} (s, a)

Sign flipped. Correct is

A^{π} (s, a) = Q^{π} (s, a) - V^{π} (s)

— "how much better than average", so a good action has positive advantage.

Why questions

WHY1. Why does taking the max over noisy estimates bias the target upward rather than downward?

Because the max preferentially selects whichever action got lucky (high-noise) this round; you systematically read the top of the noise distribution, not the middle, so

E [max] > true max

WHY2. Why does using two networks (select vs evaluate) reduce the bias?

The action chosen because its noise in $θ$ was high is evaluated by

θ^{-}

, whose noise on that action is (mostly) independent — so the lucky value isn't double-counted and gets pulled back toward truth.

WHY3. Why is the mean-centering variant preferred over max-centering for the advantage split?

Subtracting the mean is a smooth, low-variance operation; subtracting a moving

ar g max

jumps discretely as the best action changes, injecting extra variance and instability into training.

WHY4. Why does the value stream in Dueling DQN learn faster than a per-action

Q

estimate?

Every transition — no matter which action was taken — flows gradients into the shared

V (s)

stream, so state values improve even for rarely-tried actions, instead of each

(s, a)

pair being learned in isolation.

WHY5. Why does an inflated target in plain DQN keep hurting over many steps rather than washing out?

The bootstrapped backup treats the inflated value as ground truth, propagates it into neighbouring states, then re-maxes next step — so the error compounds instead of averaging away.

WHY6. Why is overestimation especially dangerous (not merely inaccurate)?

If the bias is uneven across actions, the agent can rank a genuinely bad action above a good one and choose it, degrading the policy — not just the numeric values.

WHY7. Why must we anchor

V

and

A

at all, rather than letting the network output them freely?

Because

Q = V + A

has a whole family of

(V, A)

solutions differing by a constant shift; without a constraint, gradient descent has no reason to keep

V

meaningful and the streams drift.

WHY8. Why does the parent call Double DQN a cure for "optimism" and Dueling a cure for "wastefulness"?

Optimism = the max-induced upward bias in value estimates (Double fixes the number); wastefulness = re-learning state value separately per action (Dueling fixes the architecture to share that learning).

Edge cases

EC1. If there is only

m = 1

action, how much does the max overestimate?

Zero. With one variable

max_{a} ε_{a} = ε_{1}

whose mean is

0

; the bound

σ \frac{m - 1}{2 m - 1}

gives

0

m = 1

. No selection, no bias.

EC2. In a state where all true action values are exactly equal, what does Double DQN's target look like vs plain DQN?

Plain DQN still reports

V + E [max ε] > V

; Double DQN, by evaluating with an independent net, returns a value close to the true

V

with (near) zero systematic bias.

EC3. In a Dueling net, if a state truly has identical values for all actions, what should the advantage stream output?

All-equal raw advantages, which after mean-centering become all zeros, so

Q (s, \cdot) = V (s)

for every action — exactly the "action doesn't matter here" case Dueling is built to exploit.

EC4. If

θ^{-}

were updated to equal

θ

every single step, would Double DQN still reduce bias?

Barely. With

θ^{-} = θ

selection and evaluation share the same estimator, collapsing back toward plain-DQN behaviour — the decoupling (and the stability from freezing) both need

θ^{-}

to lag.

EC5. What happens to the mean-centered aggregation when

∣ A ∣

is very large but only one action has non-zero raw advantage?

The mean

\frac{1}{∣ A ∣} \sum \tilde{A}

is tiny (one nonzero over many), so centering shifts everything by a small amount; the single action stays dominant and

V

still equals the average

Q

EC6. Double DQN with a greedy target where

θ

and

θ^{-}

happen to agree on the argmax — is any bias removed?

Little. If both nets pick the same action, evaluation still reads that action's value; the benefit of Double DQN appears precisely when the two nets disagree on the best action.

EC7. Can Dueling DQN alone fix overestimation?

No. Dueling changes the network shape but still uses a

max

-based DQN target unless you also swap in the Double target — so overestimation persists until Double is added.

Recall One-line self-test

State both fixes in one breath ::: Double = select-online, evaluate-target (kills the max's optimistic bias); Dueling = $V + (A - mean A)$ (shares state-value learning across all actions).

See also: Actor-Critic Methods (also uses an advantage) and Rainbow DQN (combines Double + Dueling + more).