Double DQN and Dueling DQN: Worked examples — AI-ML deep dive

This page drills the two ideas from Double DQN and Dueling DQN — the optimism cure (Double DQN) and the network reshaping (Dueling DQN) — through worked examples that between them touch every case class you can meet in an exam or a codebase. If a symbol here feels unfamiliar, the parent note builds it first; we only use the tools here.

Several symbols recur, so let us pin them all down before any of them appears inside an equation:

Definition Noise, spread, and averaging:

$ε_{a}$ , $σ$ , and $E [\cdot]$ The parent's overestimation bound uses three symbols we will lean on repeatedly:

$ε_{a}$ (epsilon-sub- $a$ ) = the estimation error on action $a$ : the gap between our network's noisy reading $Q (a)$ and the true value, i.e. $Q (a) = true value + ε_{a}$ . It is zero on average (sometimes too high, sometimes too low).
$σ$ (sigma) = the standard deviation of that noise: a single number saying how wide the errors typically spread. Big $σ$ = wild readings; $σ = 0$ = perfect estimates.
$E [\cdot]$ = the expectation (average) operator: $E [X]$ is the long-run average of a random quantity $X$ over many draws. So $E [max_{a} ε_{a}]$ is "the average value of the biggest error, taken over many rollouts".

With these, the parent's bound reads: for $m = ∣ A ∣$ actions, the max-operator's average upward inflation obeys $E [max_{a} ε_{a}] \leq σ \frac{m - 1}{2 m - 1}$ — it grows with $m$ .

The scenario matrix

Every example below is tagged with the cell it exercises. Read this table first — it is the map.

Cell	Case class	What is special	Example
C1	Overestimation, ordinary case	Online & target disagree on best action	Ex 1
C2	Sign flip / negative values	Best estimate is negative	Ex 2
C3	Degenerate: single action	$∣ A ∣ = 1$ , no max to inflate	Ex 3
C4	Degenerate: agreement	Online & target agree → Double = DQN	Ex 4
C5	Terminal transition	Next state does not exist ( $s^{'}$ terminal)	Ex 5
C6	Dueling: raw mean already zero	Centering changes nothing	Ex 6
C7	Dueling: raw mean non-zero	Centering absorbs the offset	Ex 7
C8	Dueling: mean vs max aggregation	Two anchors, two $V$ meanings	Ex 8
C9	Limiting behaviour: bias grows with $m$	Many actions inflate the max	Ex 9
C10	Word problem + exam twist	Combine Double + Dueling end-to-end	Ex 10

There is also one limiting edge-case that needs no separate example: the zero-discount case $γ = 0$ . When $γ = 0$ the target collapses to $y = r + 0 \cdot (bootstrap) = r$ for every variant — DQN, Double DQN, and any Dueling recombination alike. The whole $max$ /select/center machinery is multiplied away, so overestimation and identifiability both vanish; the agent becomes purely myopic (it trusts only the immediate reward). We flag this here so no reader wonders what happens at that boundary; every numbered example below uses $γ > 0$ where the machinery actually runs.

Prerequisites we lean on: Deep Q-Networks (DQN), Target Networks and Experience Replay, Advantage Function, Overestimation Bias in Q-learning.

Ex 1 — The ordinary overestimation (Cell C1)

Step 1 — Plain DQN takes the max of the target net. $max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'}) = max (- 0.2, 0.4, - 0.1) = 0.4.$ Why this step? Plain DQN uses one network ( $θ^{-}$ ) for both choosing and scoring. It grabs the largest target number — which is exactly the one whose noise $ε_{a}$ was luckiest.

Step 2 — Form the DQN target. $y^{D QN} = r + γ \cdot 0.4 = 0 + 0.9 (0.4) = 0.36.$ Why this step? The Bellman target is "reward now, plus discounted value of the best next move". We plug the max from Step 1 into $r + γ (\cdot)$ to get the single number the network trains toward.

Step 3 — Double DQN: select with online, evaluate with target. $ar g max_{a^{'}} Q_{θ} (s^{'}, a^{'}) = a_{1} (since 0.3 is largest in the online list) .$ $Q_{θ^{-}} (s^{'}, a_{1}) = - 0.2.$ Why this step? The online net's lucky action $a_{1}$ has noise independent of the target net's reading. We do not get to reuse the same lucky spoon twice.

Step 4 — Form the Double DQN target. $y^{D o u b l eD QN} = 0 + 0.9 (- 0.2) = - 0.18.$ Why this step? Same Bellman shape $r + γ (\cdot)$ as Step 2, but now the discounted term is the decoupled value $- 0.2$ from Step 3 — this is precisely where the two methods diverge in their answer.

Verify: truth is $0$ . DQN overshoots to $+ 0.36$ ; Double lands at $- 0.18$ . $∣ - 0.18 ∣ < ∣ 0.36 ∣$ , so Double is closer to $0$ . The optimism was cut. ✔

Read the figure: the cyan bars are the online net, the amber bars the target net. Notice DQN's amber arrow points at the tallest amber bar ( $a_{2} = 0.4$ ), while Double's cyan arrow first picks the tallest cyan bar ( $a_{1}$ ) and then reads the amber value sitting under $a_{1}$ ( $- 0.2$ ). The picture makes visible that the two methods read different cells of the target row.

Ex 2 — When the best estimate is negative (Cell C2)

Step 1 — DQN max over target. $max (- 0.9, - 0.6, - 0.3) = - 0.3, y^{D QN} = 0.9 (- 0.3) = - 0.27.$ Why this step? "Max" means largest, even among negatives — $- 0.3$ beats $- 0.9$ . The optimism bias survives sign changes; it always favours the luckiest reading.

Step 2 — Double selects with online. $ar g max_{a^{'}} Q_{θ} = a_{2} (since - 0.4 is the largest online value) .$ Why this step? Selection in Double DQN always uses the online net, regardless of sign. Among the online numbers $- 1.0, - 0.4, - 0.7$ the largest is $- 0.4$ at $a_{2}$ , so the online net's chosen action is $a_{2}$ — and crucially this is a different action from the target's own favourite $a_{3}$ .

Step 3 — Evaluate $a_{2}$ with target. $Q_{θ^{-}} (s^{'}, a_{2}) = - 0.6, y^{D o u b l eD QN} = 0.9 (- 0.6) = - 0.54.$ Why this step? Online and target disagree on which action is best ( $a_{2}$ vs $a_{3}$ ). Double honours the online's choice $a_{2}$ and reads its target value, refusing to grab the single luckiest cell.

Verify: $- 0.54 < - 0.27$ . Double gives the lower (less optimistic) target even in fully negative territory — exactly the intended direction of the correction. ✔

Ex 3 — Degenerate: only one action (Cell C3)

Step 1 — The max over one element is that element. $max Q_{θ^{-}} = 0.5, ar g max = a_{1} .$ Why this step? From the parent's bound $E [max_{a} ε_{a}] \leq σ \frac{m - 1}{2 m - 1}$ : at $m = 1$ the numerator $m - 1 = 0$ , so the whole bound is $0$ and the bias is exactly zero. There is no "lucky pick" when there is nothing to pick from.

Step 2 — Both targets. $y^{D QN} = 0.9 (0.5) = 0.45, y^{D o u b l eD QN} = 0.9 Q_{θ^{-}} (a_{1}) = 0.9 (0.5) = 0.45.$ Why this step? Selection and evaluation are forced onto the same and only action $a_{1}$ , so both methods read the identical target value $0.5$ and pass it through the same $r + γ (\cdot)$ — there is no room for them to diverge.

Verify: identical. Double DQN reduces to DQN whenever there is one action — the cure and disease coincide because there is no over-optimism to cure. ✔

Ex 4 — Degenerate: the networks agree (Cell C4)

Step 1 — DQN max. $max Q_{θ^{-}} = 0.8$ at $a_{2}$ . Why this step? Plain DQN reads the largest target value; among $0.3, 0.8, 0.2$ that is $0.8$ at $a_{2}$ . We record both the value and the action so we can compare against the online net's choice next.

Step 2 — Double selection. $ar g max Q_{θ} = a_{2}$ — same action. Why this step? Double selects with the online net; its largest value $0.9$ also sits at $a_{2}$ . Because both nets nominate the very same action, the decoupling has nothing to separate.

Step 3 — Double evaluation. $Q_{θ^{-}} (a_{2}) = 0.8$ . Why this step? When selection and evaluation land on the same cell, Double reads exactly the value DQN read ( $0.8$ ). The decoupling only bites when the two nets disagree.

Step 4 — Targets. $y^{D QN} = 1 + 0.9 (0.8) = 1.72, y^{D o u b l eD QN} = 1 + 0.9 (0.8) = 1.72.$ Why this step? Both methods feed the identical discounted value $0.8$ into $r + γ (\cdot)$ with $r = 1$ , so they must produce the identical target $1.72$ .

Verify: both $1.72$ . Double DQN is a soft correction — it never harms the agreeing case, it only trims the disagreeing (biased) ones. ✔

Ex 5 — Terminal transition (Cell C5)

Step 1 — Terminal means no future. For a terminal $s^{'}$ there is no next action and no bootstrap. The Bellman target collapses: $y = r + γ \cdot 0 = r .$ Why this step? $s^{'}$ being terminal is coded by a done-flag $d = 1$ ; the general target is $y = r + γ (1 - d) [bootstrap]$ . With $d = 1$ the whole $γ (\dots)$ block is multiplied by $0$ .

Step 2 — Plug in. $y = 2 + 0.9 (1 - 1) (anything) = 2.$ Why this step? We substitute $r = 2$ , $γ = 0.9$ , $d = 1$ ; the $(1 - 1) = 0$ factor annihilates whatever bootstrap value would have sat there, leaving only $r$ .

Verify: $y = 2 = r$ . Overestimation cannot occur at a terminal step because there is no max to inflate — Double DQN and DQN are trivially identical here, and both equal the raw reward. ✔

Ex 6 — Dueling, raw mean already zero (Cell C6)

Step 1 — Compute the mean of $\tilde{A}$ . $\overset{ˉ}{A} = \frac{1}{3} (1 + (- 1) + 0) = 0.$ Why this step? Mean-centering subtracts $\overset{ˉ}{A}$ from every advantage, so we must first know $\overset{ˉ}{A}$ . Averaging the three raw advantages gives $0$ here.

Step 2 — Center. $A = \tilde{A} - \overset{ˉ}{A} = [1, - 1, 0]$ (unchanged). Why this step? The mean-centering step $A = \tilde{A} - \frac{1}{∣ A ∣} \sum \tilde{A}$ is what pins $V$ to $mean (Q)$ (see the parent's aggregation formula). If the mean is already $0$ , it is a no-op — but we still must run it so behaviour is uniform across states.

Step 3 — Recombine. $Q = V + A = 5 + [1, - 1, 0] = [6, 4, 5] .$ Why this step? The dueling aggregation is $Q (s, a) = V (s) + A (s, a)$ with the centered advantages; adding $V = 5$ to each centered advantage produces the per-action $Q$ values.

Verify: $mean (Q) = (6 + 4 + 5) /3 = 5 = V$ . The anchor holds; the best action is $a_{1}$ with advantage $+ 1$ . ✔

Ex 7 — Dueling, raw mean non-zero (Cell C7)

Step 1 — Mean of raw advantages. $\overset{ˉ}{A} = (3 + 1 + 2) /3 = 2$ . Why this step? Again centering needs $\overset{ˉ}{A}$ first; here the raw advantages average to $2$ , a non-zero offset that centering must strip out.

Step 2 — Center. $A = [3 - 2, 1 - 2, 2 - 2] = [1, - 1, 0]$ . Why this step? Centering strips the shared offset $+ 2$ out of the advantage stream. Advantages are relative quantities — only their differences carry meaning — so removing a constant is legitimate.

Step 3 — Correct recombination. $Q = 5 + [1, - 1, 0] = [6, 4, 5], mean (Q) = 5 = V ✔.$ Why this step? Adding $V = 5$ to the centered advantages keeps $mean (Q)$ pinned to $V$ , exactly as the aggregation formula guarantees.

Step 4 — The naïve (wrong) version. $Q^{nai v e} = V + \tilde{A} = 5 + [3, 1, 2] = [8, 6, 7] .$ Why this step? We deliberately skip centering to expose the failure: the $+ 2$ offset leaks into $Q$ , giving $mean (Q^{nai v e}) = 7 \neq = 5 = V$ . The value stream has silently inflated by $2$ , and the split $V, A$ is no longer identifiable (the parent's identifiability trap).

Verify: centered $mean (Q) = 5 = V$ ✔; naïve $mean (Q^{nai v e}) = 7 \neq = 5$ — anchor broken, confirming why centering is mandatory. ✔

Read the figure: the cyan bars (correctly centered $Q = [6, 4, 5]$ ) hug the dashed white $V = 5$ line — their average lands exactly on it. The amber bars (naïve $Q = [8, 6, 7]$ ) float a full $+ 2$ higher, their average marked by the amber dotted line at $7$ . The picture shows the offset physically "lifting" the whole $Q$ row away from $V$ when centering is skipped.

Ex 8 — Mean vs max aggregation (Cell C8)

Step 1 — Mean anchor (from Ex 6/7). $Q^{m e an} = 5 + [1, - 1, 0] = [6, 4, 5], V = mean (Q^{m e an}) = 5.$ Why this step? This is the standard mean-centered aggregation; we restate its result so we have a baseline to compare the max anchor against.

Step 2 — Max anchor. $max_{a^{'}} A = max (1, - 1, 0) = 1$ , so subtract $1$ from each advantage: $Q^{ma x} = 5 + ([1, - 1, 0] - 1) = 5 + [0, - 2, - 1] = [5, 3, 4] .$ Why this step? The max anchor pins $V = max_{a} Q$ : subtracting $max_{a^{'}} A$ forces the best action to advantage $0$ , so its $Q$ equals $V$ exactly. Here $max (Q^{ma x}) = 5 = V$ .

Step 3 — Compare interpretations.

Mean anchor: $V = mean (Q) = 5$ ("how good is this state on average").
Max anchor: $V = max_{a} Q = 5$ ("how good is this state if I play optimally").

Why this step? The two anchors do not just produce different $Q$ vectors — they redefine what the scalar $V$ means, which is why the choice of anchor is a modelling decision, not a cosmetic one.

Verify: $mean (Q^{m e an}) = 5$ ✔ and $max (Q^{ma x}) = 5$ ✔. Both anchor to $5$ here by coincidence of numbers, but their meanings differ. The mean version is preferred in practice: subtracting a fixed mean is lower-variance than chasing a moving $ar g max$ . ✔

Ex 9 — Limiting behaviour: bias grows with number of actions (Cell C9)

Step 1 — Evaluate the bound at each $m$ . $m = 1 : 1 \cdot \frac{0}{1} = 0.$ $m = 2 : 1 \cdot \frac{1}{3} \approx 0.5774.$ $m = 4 : 1 \cdot \frac{3}{7} \approx 1.1339.$ $m = 8 : 1 \cdot \frac{7}{15} \approx 1.8074.$ Why this step? Each number is the worst-case average upward inflation $E [max_{a} ε_{a}]$ of the DQN target from the $max$ operation alone, holding truth at $0$ and $σ = 1$ . It is the size of the disease Double DQN treats.

Step 2 — Read the trend. $0 < 0.58 < 1.13 < 1.81$ — monotonically increasing. Why this step? Lining the four values up shows the bias is not constant: doubling the action count keeps pushing the ceiling higher, which is the whole point of the cell.

Verify: at $m = 1$ the bound is exactly $0$ (matches Ex 3's single-action result), and it climbs with $m$ . So DQN's overestimation is worst in large-action-space environments — precisely where Double DQN pays off most. ✔

Read the figure: the cyan curve is the bound plotted against the action count $m$ ; the amber-filled dots mark the four cases we computed. See how it starts pinned at $0$ for $m = 1$ (a single action, no bias) and rises steadily — the visual proof that "more actions ⇒ more optimism to cure".

Ex 10 — Word problem + exam twist: full Double-Dueling pass (Cell C10)

Step 1 — Build online $Q$ (center its advantages). $\overset{ˉ}{A}_{θ} = (0.2 + 0.5 + 0.2) /3 = 0.3, A_{θ} = [- 0.1, 0.2, - 0.1] .$ $Q_{θ} = 10 + [- 0.1, 0.2, - 0.1] = [9.9, 10.2, 9.9] .$ Why this step? Selection in Double DQN uses the online net, and this net is dueling — so we must reconstruct $Q$ from its two streams (mean-centered, as in Ex 6/7) before taking any argmax.

Step 2 — Online selection (the argmax on $Q_{θ}$ ). $ar g max_{a^{'}} Q_{θ} (s^{'}, a^{'}) = a_{2} ("straight", Q_{θ} = 10.2) .$ Why this step? Double DQN's selection rule says the action is whichever gives the largest online $Q$ . Among $9.9, 10.2, 9.9$ that is $10.2$ at $a_{2}$ , so the online net picks "straight". This is the action we will hand to the target net.

Step 3 — Build target $Q$ (center its advantages). $\overset{ˉ}{A}_{θ^{-}} = (0.1 + 0.0 + 0.3) /3 = 0.1 \overline{3}, A_{θ^{-}} = [- 0.0 \overline{3}, - 0.1 \overline{3}, 0.1 \overline{6}] .$ $Q_{θ^{-}} = 9 + A_{θ^{-}} = [8.9 \overline{6}, 8.8 \overline{6}, 9.1 \overline{6}] .$ Why this step? The evaluation uses the target net, which is also dueling, so we must reconstruct its $Q$ from its own two streams the same way before we can read a value off it.

Step 4 — Evaluate the online-selected action $a_{2}$ with the target net. $Q_{θ^{-}} (s^{'}, a_{2}) = 9 - 0.1 \overline{3} = 8.8 \overline{6} = \frac{133}{15} .$ Why this step? Double DQN evaluates the online's choice $a_{2}$ using the target net — not the target's own max. The target's own best is $a_{3}$ ("right", $Q = 9.1 \overline{6}$ ), so the two nets disagree; Double honours the online's pick and reads its (lower) target value, dodging the optimism.

Step 5 — Form the target. $y^{D o u b l eD u e l in g} = r + γ Q_{θ^{-}} (s^{'}, a_{2}) = 1 + 0.9 (\frac{133}{15}) = 1 + \frac{119.7}{15} = 8.98.$ Why this step? The final step is the same Bellman shape $r + γ (\cdot)$ used everywhere on this page, now fed the decoupled value $8.8 \overline{6}$ from Step 4 with $r = 1, γ = 0.9$ — giving the single number the network trains toward.

Verify: exam-twist answer — online selects $a_{2}$ ("straight"), target's own argmax is $a_{3}$ ("right"), so they disagree; Double DQN uses $a_{2}$ 's target value $8.8 \overline{6}$ giving $y = 8.98$ . And note the dueling payoff: because $V = 10$ dominates and the advantages are tiny ( $\pm 0.2$ ), all three $Q$ values sit within $0.3$ of each other — confirming "steering barely matters", and the shared $V$ stream learned this state's worth from every transition. ✔

Recall Which cells did we cover?

Ordinary bias (C1) ::: Ex 1 Negative-valued state (C2) ::: Ex 2 Single action degenerate (C3) ::: Ex 3 Networks agree degenerate (C4) ::: Ex 4 Terminal transition (C5) ::: Ex 5 Dueling raw-mean-zero (C6) ::: Ex 6 Dueling raw-mean-nonzero (C7) ::: Ex 7 Mean vs max anchor (C8) ::: Ex 8 Bias grows with m (C9) ::: Ex 9 Word problem + twist, full pass (C10) ::: Ex 10 Zero-discount edge-case (gamma=0) ::: covered in the matrix note (target collapses to r)

Related deep vaults: Actor-Critic Methods (where the Advantage Function returns as a first-class citizen) and Rainbow DQN (which stacks Double + Dueling with more tricks). See also Overestimation Bias in Q-learning for the statistical root.