Foundations — Double DQN and Dueling DQN

Before you can read a single formula in Double DQN and Dueling DQN, you need the alphabet. Every letter below is defined in plain words, drawn as a picture, and justified — why does the topic even need this symbol? Read top to bottom; each rung stands on the one before it.

1. The state $s$ — "where am I right now?"

Picture a single frame of a video game frozen on the screen. That frozen frame is $s$. Nothing about the future or the past — just "here is the world, right now".

Why the topic needs it: every value the agent learns is conditioned on a state. There is no "how good is this action" without first asking "in which situation?".

2. The action $a$ — "what can I do?"

Why the topic needs it: Dueling DQN's whole trick is about how many buttons matter. The symbol $|\mathcal{A}|$ appears directly in its mean-centering formula (you divide by it), so you must be comfortable that it is just "count the buttons".

3. The reward $r$ — "did that go well?"

Picture a scoreboard that ticks up or down by one number each time you press a button. That tick is $r$.

Why the topic needs it: $r$ is the only real, non-guessed quantity in every target formula. Everything else is an estimate; $r$ is ground truth the environment gives us.

4. The return and the discount $\gamma$ — "total future reward, but nearer counts more"

Why the shrinking? Two reasons. (1) The future is uncertain — a reward you might get in 50 steps is less trustworthy than one now. (2) The sum must stay a finite number even over infinitely many steps; multiplying by $\gamma <1$ each time makes it converge.

Why the topic needs it: $\gamma$ sits right in front of the estimated future value in every target — $y=r+\gamma (\dots )$. The entire "overestimation" disease lives inside that $\gamma (\dots )$ term.

5. The Q-value $Q(s,a)$ — "score of doing $a$ in $s$"

Why the topic needs it: Double DQN worries that these $Q$ numbers come out too big. Dueling DQN worries that we store them wastefully. You cannot understand either fix without knowing $Q$ is "the score of a (situation, choice) pair."

6. The network and its weights $\theta$ — "the $Q$-table is too big, so we approximate it"

Why the topic needs it: every formula tags $Q$ with a subscript — $Q_{\theta }$ or $Q_{{\theta }^{-}}$ — and those subscripts are the entire point of Double DQN. If you don't read $\theta$ as "which set of dials", the fix looks like meaningless decoration.

7. The target network ${\theta }^{-}$ — "a frozen photocopy of the dials"

Why the topic needs it: Double DQN's slogan is literally "select with $\theta$, evaluate with ${\theta }^{-}$". Two symbols, two networks. If ${\theta }^{-}$ is a mystery, the cure is a mystery.

8. $\max$, $\arg \max$, and the difference between them

This is the single most confused pair in the whole topic, so we separate them slowly.

Why the topic needs it — and why the split matters: plain DQN does ${\max }_a{Q}_{{\theta }^{-}}$: it uses one network to both pick the winner and read its value. Double DQN carefully separates the two: $\arg \max$ (pick) with $\theta$, then read the value with ${\theta }^{-}$. You literally cannot state the fix unless you know that $\arg \max$ gives an action and $\max$ gives a value. This is the crux of the whole page — see Overestimation Bias in Q-learning.

9. Expectation $\mathbb{E}[\cdot ]$ and noise $\epsilon$ — "on average, over the randomness"

Why the topic needs it: the key inequality $\mathbb{E}[{\max }_a{\epsilon }_a]\ge 0$ is what proves DQN is an optimist. It's an average-of-noise statement, pure and simple.

The picture above is the whole disease in one image: each guess is truth-plus-noise, and picking the maximum systematically grabs whichever one got lucky — so the max floats above the true value.

10. Value $V(s)$ and Advantage $A(s,a)$ — "state goodness vs. action edge"

Why the topic needs it: Dueling DQN builds two separate streams — one computes $V$, one computes $A$ — and glues them back with $Q=V+(A-\text{mean }A)$. Without knowing $V$ is "state goodness" and $A$ is "edge over average", that formula is gibberish.

Prerequisite map

Equipment checklist

Cover the right side and answer out loud before revealing.