Double DQN and Dueling DQN: Visual walkthrough

Step 0 — The three words we keep saying

Before any maths, three plain-word anchors. Every symbol below is earned here.

The whole drama on this page is: what happens when you take the biggest of several wobbly guesses.

Step 1 — Draw the wobble: what "noisy estimate" means

WHAT. Suppose three actions are actually equally good. Call that shared true value $V$ . Our network cannot see $V$ ; it sees $Q (a_{i}) = V + ε_{i}$ .

$V$ :: the true value (a fixed height, same for all three actions).
$ε_{i}$ (Greek "epsilon", the noise on action $i$ ) :: a small random up/down wobble, sometimes $+$ , sometimes $-$ , averaging to zero.

WHY. We must model noise because a freshly-training network is always wrong by some random amount. If we pretend estimates are exact, we miss the whole disease.

PICTURE. Three actions sit at the same true height (dashed line). The dots are the wobbled guesses — some float above the line, some below.

Step 2 — Take the max, and watch it float upward

WHAT. The DQN target uses $max_{a} Q (s, a)$ — the biggest of the wobbled dots.

$max_{a}$ :: "pick the largest over all actions $a$ ". It reads off the top dot in Step 1.

WHY this operation at all? DQN wants the value of acting optimally next step, and "optimal" means "whichever action scores highest". So the max is not a bug — it is the definition of "play the best move". The bug is that we take the max of guesses, not of truths.

PICTURE. The max reaches for whichever action got the luckiest upward wobble. So the chosen height sits above the true dashed line, on average.

Step 3 — Why "average of the max" beats "max of the average"

WHAT. We compare two quantities that look similar but are not: $E [max_{a} ε_{a}]$ versus $max_{a} E [ε_{a}]$ .

WHY. This single inequality is the whole disease, stated exactly.

$max_{a} E [ε_{a}]$ :: first average each action's wobble (each averages to $0$ ), then take the biggest → biggest of ${0, 0, 0} = 0$ . No bias.
$E [max_{a} ε_{a}]$ :: first take the biggest wobble each roll, then average → a positive number, because you keep grabbing whichever happened to be highest.

PICTURE. Left panel: many re-rolls, each time we mark the top dot; those top marks pile up above zero. Right panel: the averaged (settled) dots all sit at zero. The gap between the two piles is the bias.

Formula How big is the gap? (a derivable bound)

For $m$ i.i.d. zero-mean wobbles of standard deviation $σ$ , a classical result bounds the expected max: $E [max_{a} ε_{a}] \leq σ \frac{m - 1}{2 m - 1} .$

$σ$ (sigma) :: the size of the wobble (its standard deviation). Bigger noise → bigger lie.
$m = ∣ A ∣$ :: number of actions.

Where it comes from (sketch). Start from the variance of the centred max, $E [(max_{a} ε_{a})^{2}] \leq σ^{2}$ contributions across $m$ independent variables; a standard second-moment (Cauchy–Schwarz style) argument on the order statistics of $m$ i.i.d. zero-mean variables yields exactly the factor $\frac{m - 1}{2 m - 1}$ . It is not something to memorise — the shape is the message:

at $m = 1$ it is $0$ (no others to beat),
it grows without bound as $m$ increases (more buttons ⇒ more chances for a lucky one),
it scales with $σ$ (noisier estimates ⇒ bigger overestimate).

Step 4 — The degenerate check: when is there NO bias?

WHAT. Every good derivation must survey its edge cases. Two make the bias vanish:

$m = 1$ (one action). Then $max$ has nothing to choose — the plug-in of $m = 1$ into the bound gives $σ \cdot 0/ 1 = 0$ . No bias.
$σ = 0$ (no noise, perfect estimates). Then $ε_{a} = 0$ for all $a$ , so $max_{a} ε_{a} = 0$ . No bias.

WHY show this. It proves the bias is caused by noisy estimates over multiple choices — not by the max being evil in itself. Remove either ingredient and the disease disappears.

PICTURE. Left: single action, the "max" is just the one dot — no upward pull. Right: zero-noise case, all dots collapse onto the true line — the max sits exactly on it.

Step 5 — The fix: let two different friends choose and score

WHAT. The bias came from one noisy network doing two jobs: pick the top action and report its value. If action $a^{⋆}$ was picked because its wobble was high, then reading its value re-uses that same high wobble — we lock the luck in.

The fix: use two networks whose estimation errors are (approximately) independent.

Online network $θ$ (theta) → does the selection ("which button?").
Target network $θ^{-}$ (theta-minus, the frozen copy from Target Networks and Experience Replay) → does the evaluation ("what's it worth?").

WHY it works. The action $θ$ loves may have gotten a lucky wobble in $θ$ . But $θ^{-}$ 's wobble on that same action is a fresh, independent roll — most likely ordinary, not lucky. So we no longer double-count the luck.

PICTURE. Two boards side by side. On board $θ$ the top action is circled (selection). We carry that same action index to board $θ^{-}$ and read its dot there — which sits near the true line, not at the peak.

Step 6 — Now the second disease: the split $Q = V + A$ is ambiguous

WHAT. Switch to Dueling. We want the network to output a state value $V (s)$ ("is this spot good?") and per-action advantages $A (s, a)$ ("how much better than average is button $a$ ?"), then combine: $Q = V + A$ . See the Advantage Function.

$V (s)$ :: one number per state — the average quality of the state.
$A (s, a)$ :: one number per action — its edge over the average.

WHY there's a problem. Add any constant $c$ to $V$ and subtract $c$ from every $A$ — the sum $Q$ is identical. The network can't tell the "right" split from a shifted one; $V$ drifts and loses meaning. This is the identifiability trap.

PICTURE. Two different $(V, A)$ splits (one shifted up by $c$ ) produce the exact same $Q$ bars. The network sees no difference — so it can't pin $V$ .

Step 7 — The anchor: subtract the mean advantage

WHAT. Force the advantages to sum to zero across actions. The network's raw advantage stream outputs $\tilde{A}$ (A-tilde); we build the anchored advantage by subtracting its mean, and combine that with $V$ .

$\tilde{A} (s, a)$ (A-tilde) :: the raw, un-anchored advantage straight from the network.
$∣ A ∣ = m$ :: the number of actions.
$\frac{1}{∣ A ∣} \sum_{a^{'}} \tilde{A} (s, a^{'})$ :: the average of the raw advantages.

WHY it fixes it. After centering, the advantages sum to zero always. Summing the combine-equation over all actions then forces $V (s) = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)$ — $V$ is pinned to the average $Q$ . One unique split survives.

PICTURE. The raw advantages (some non-zero mean) get slid down so their average hits zero; the removed offset lands cleanly in $V$ , and the recovered $V$ equals the mean height of the final $Q$ bars.

The one-picture summary

Both diseases, both cures, on one board:

Left (Double): noisy max floats up → split the "choose" and "score" jobs across $θ$ and $θ^{-}$ (whose errors are independent) → the score drops back toward truth.
Right (Dueling): ambiguous $V + A$ → subtract the mean raw advantage → $V$ pinned to $mean (Q)$ .

These are orthogonal — a Dueling architecture trained with a Double target is the standard "Double Dueling DQN", itself a component of Rainbow DQN. The advantage idea also links to Actor-Critic Methods, where a critic estimates $V$ and the actor uses $A$ -like signals.

Recall Feynman retelling — the whole walkthrough in plain words

You are scoring ice-cream shops with noisy tiny spoon-tastes, where each flavour's luck is independent of the others. Double: if you both pick the flavour and score it with the same lucky spoon, a bad flavour that happened to taste great fools you — and this always pulls scores upward (the max grabs the luckiest wobble; its average is at least zero because the max is never below any single flavour). Fix: one friend picks the flavour, a different friend — whose spoon-luck is unrelated — tastes and scores it. The lucky spoon can't fool two independent tongues. Dueling: instead of scoring every flavour from scratch, first ask "is this shop good overall?" ( $V$ ) then "is chocolate a bit better or worse than average here?" ( $A$ ). But " $V$ plus $A$ " is ambiguous — you could pretend the shop is amazing and every flavour is a bit disappointing, and get the same totals. Fix: force the little better/worse numbers to average out to zero, so the "shop goodness" number is forced to equal the average of the flavour scores. Now it means exactly one thing.

Recall

Why is $E [max_{a} ε_{a}] \geq 0$ when each $ε_{a}$ averages to zero? ::: Because $max_{a} ε_{a} \geq ε_{1}$ on every roll, and monotonicity of expectation turns that into $E [max_{a} ε_{a}] \geq E [ε_{1}] = 0$ . What assumption on the noise makes the overestimation argument work? ::: The wobbles are i.i.d. — independent and identically distributed (same spread, no conspiring). In Double DQN, which network selects and which evaluates, and why must their errors be independent? ::: Online $θ$ selects the argmax action; target $θ^{-}$ evaluates it; independence stops the lucky pick from being confirmed twice. What does subtracting the mean advantage pin $V$ to? ::: The average of $Q$ across actions, $V = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)$ . Two settings where overestimation bias vanishes? ::: A single action ( $m = 1$ ), or zero estimation noise ( $σ = 0$ ).