Foundations — Deep Q-Networks (DQN)

Before you can read the parent note, you must be able to read its symbols out loud in plain English. This page is a dictionary — but a dictionary where every word comes with a picture and a reason it exists. We build strictly bottom-up: nothing appears until the thing it leans on is already defined.

1. The stage: state, action, reward

Everything in reinforcement learning happens as a loop between two characters: an agent (the learner) and an environment (the world it lives in).

Look at the loop above: the agent sees $s$, picks $a$, the world replies with a reward $r$ and a new state $s^{\prime }$ (read "s-prime" — the state that comes next). That primed symbol is used constantly in the parent, so lock it in:

One full hop $(s,a,r,s^{\prime })$ — "I was here, did this, got this, ended up there" — is called a transition. Remember that 4-tuple; the whole replay buffer is a pile of them.

2. Following a rule: the policy $\pi$

3. Caring about the future: discount $\gamma$

A reward now and a reward in a thousand steps are not worth the same. We shrink far-off rewards with a discount factor.

Why ${\gamma }^k$ and not, say, subtracting a constant each step? Because multiplying repeatedly is what "compounding decay" looks like — the same shape as radioactive half-life or interest. Each extra step of delay multiplies the weight by another $\gamma$, so a reward $k$ steps out is worth ${\gamma }^k$ of its face value. Two cases to feel:

• $\gamma =0$: only the immediate reward matters — a total short-term thinker.
• $\gamma \to 1$: the distant future matters almost as much as now — a patient planner.

DQN typically uses $\gamma =0.9$ or $0.99$.

4. The star of the show: the value $Q(s,a)$

Now we can build the single most important symbol.

The full formula in the parent is a sum with an $\mathbb{E}$ in front and a $\Sigma$ inside. Let's decode those two symbols so they stop being scary.

So the parent's line reads, in English: "$Q$ is the average, over all the randomness, of the sum of all future rewards, each discounted by how far away it is."

$Q^{\pi }(s,a)=\mathbb{E}\left[{\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}|s_t=s,a_t=a\right]$

The vertical bar $\mid$ reads "given that" — given we start in state $s$ and take action $a$ at time $t$.

5. The best possible plan: $Q^{\ast }$ and $\max$

This is exactly why the parent's Bellman Equation uses ${\max }_{a^{\prime }}$: it assumes optimal play from the next state onward. And here is the hidden restriction that trips people up: to take a $\max$ you must be able to list all the actions. That's why DQN handles only discrete action sets — you can't loop over infinitely many continuous choices.

6. The neural network: $\theta$ and $Q(s,a;\theta )$

A table can't hold a value for every possible screen (there are astronomically many). So DQN replaces the table with a function that computes the value.

7. Learning signals: TD error and the loss $\mathcal{L}$

8. Where the data lives: replay buffer $D$ and $ϵ$

How the pieces feed the topic

Equipment checklist

Read each cue, answer in your head, then reveal.

When every reveal feels obvious, you're ready for Deep Q-Networks (DQN) itself.