Deep Q-Networks (DQN): Question bank — AI-ML deep dive

The building blocks are Q-Learning, the Bellman Equation, Experience Replay, and Temporal Difference Learning. This page is self-contained: the symbol reminder below defines every piece of notation you will meet, so you never have to leave to answer a question.

Definition Symbol reminder — read once, refer back anytime

$s, a, r, s^{'}$ — the state you are in, the action you take, the reward you get, and the next state you land in. One "transition" is the tuple $(s, a, r, s^{'})$ .
$Q (s, a; θ)$ — the online network's guess of how good taking action $a$ in state $s$ is, in units of expected future reward. $θ$ (theta) are the weights of that network — the numbers gradient descent adjusts.
$θ^{-}$ (theta-minus) — a frozen older copy of $θ$ , the target network. It is refreshed by $θ^{-} \leftarrow θ$ only every $C$ steps.
$γ$ (gamma), $0 \leq γ < 1$ — the discount factor: how much a reward one step later is worth compared to now. $γ$ near $0$ = short-sighted, near $1$ = far-sighted.
$y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ — the target (label) we regress toward: immediate reward plus discounted value of the best next action $a^{'}$ .
$δ = y - Q (s, a; θ)$ — the TD error: how far our current guess is from the target.
$ϵ$ (epsilon) — probability of acting randomly instead of greedily (exploration).

The picture below fixes these symbols visually before you start — the online net and its frozen twin feeding one squared-error arrow.

True or false — justify

TF1. A neural network in DQN stores one weight per state, just like a table stores one number per state.

False. The whole point is that the ==network shares weights

θ

across states==, so learning about one screen generalises to similar screens — a table cannot do that.

TF2. DQN is on-policy because it acts with the same

ϵ

-greedy policy it is training.

False. DQN is off-policy: it acts

ϵ

-greedily but the target uses

max_{a^{'}} Q

, i.e. it learns the value of the greedy policy, not the behaviour policy.

TF3. If the loss

L (θ)

reaches exactly zero on the buffer, the agent has found the optimal policy.

False. Zero loss only means

Q

satisfies Bellman on the sampled transitions with the current bootstrapped targets; targets can still be biased (e.g. overestimation) or the buffer may not cover the state space.

TF4. Because

γ < 1

, the infinite reward sum in

Q^{π}

is guaranteed finite for bounded rewards.

True. With rewards bounded by

R

, the sum is at most

R \sum_{k} γ^{k} = R / (1 - γ)

, so discounting keeps the value finite — this is why

γ

exists.

TF5. Setting

γ = 1

removes the contraction that makes bootstrapped training stable.

True. At

γ = 1

the Bellman update is no longer a strict contraction mapping, so errors need not shrink each step and the target becomes sensitive to episode length — that is why we keep

γ < 1

TF6. The target network and the online network must eventually converge to identical weights.

True in spirit. Since

θ^{-} \leftarrow θ

every

C

steps, once

θ

stops changing the copies coincide; the freezing only delays

θ^{-}

, it does not make it a permanently different function.

TF7. Experience replay is useless once the agent is good, because old data was collected by a bad policy.

False. Q-learning is off-policy, so old transitions remain valid Bellman samples; reusing them is precisely where DQN's sample efficiency comes from.

TF8. The TD error

δ = y - Q (s, a; θ)

being negative means the agent made a mistake in the environment.

False. Negative

δ

just means the current estimate overshoots the bootstrapped target, so we nudge

Q (s, a)

down — it is about estimate calibration, not about a wrong action.

Spot the error

SE1. "To compute the target

y

, differentiate through

Q (s^{'}, a^{'}; θ^{-})

so gradients flow into both terms."

The target is treated as a fixed label (detached). Backpropagating through it turns a fixed-point equation into a self-referential moving target and diverges — that is why the target network exists.

SE2. "For a terminal transition,

y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})

where

s^{'}

is the terminal state."

Wrong. At a terminal there is ==no future reward and no valid

s^{'}

to evaluate==, so

y = r

; the

γ max Q

term must be dropped.

SE3. "DQN handles a robot arm with continuous torque because it handles continuous Atari-pixel states."

Continuous states are fine, but

max_{a^{'}}

requires enumerating actions, so actions must be discrete. Continuous actions need DDPG/SAC or Policy Gradient Methods.

SE4. "We use the same network for choosing and evaluating the next action, and that's optimal."

Using one network to both pick and value the max causes overestimation bias (positive noise gets selected then trusted). Double DQN fixes this by splitting selection and evaluation.

SE5. "Keep the replay buffer tiny and only recent, so training stays relevant."

A tiny recent buffer reintroduces correlation and forgetting — the very problem replay solves. Off-policy validity means old data is still usable; a large buffer helps.

SE6. "Anneal

ϵ

up over time so the agent explores more as it learns."

Backwards. We ==anneal

ϵ

down== (e.g.

1 \to 0.1

): explore heavily early when nothing is known, exploit late once

Q

is trustworthy. See Epsilon-Greedy Exploration.

SE7. "Since the target uses

θ^{-}

, we should also act using

θ^{-}

No. The target network is ==only for computing

y

==; action selection uses the current online net

θ

, which is more up to date.

SE8. "The Bellman split

Q^{*} (s, a) = E [r + γ max_{a^{'}} Q^{*} (s^{'}, a^{'})]

works because we assume the next action is also

a

No. The tail is evaluated under the ==best next action

a^{'}

== via the

max

, not by repeating

a

; that recursion is what makes it optimality, not just policy evaluation.

Why questions

W1. Why bootstrap with our own estimate

max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})

instead of the true

Q^{*}

Because ==

Q^{*}

is unknown==; bootstrapping plugs in our current best guess of the next state's value, which is what Temporal Difference Learning does — learn a guess from a guess.

W2. Why does freezing the target for

C

steps stabilise training?

It stops the regression target from moving while we chase it. A target that shifts every gradient step is like a dog chasing its tail — the fixed target gives supervised-learning-like stability.

W3. Why does random minibatch sampling matter for SGD here, beyond "breaking correlation"?

SGD's convergence guarantees assume unbiased gradient estimates with bounded variance; correlated consecutive frames make each minibatch gradient a biased, high-variance estimate of the true loss gradient, so steps point the wrong way. Random sampling from the buffer restores near-i.i.d. batches, reducing gradient variance and letting the averaged updates actually descend the loss — plus it recycles rare transitions so their signal is not seen once and forgotten.

W4. Why is DQN's ability to generalise both its strength and a source of instability?

Weight sharing means updating one state also shifts nearby states' Q-values, including the bootstrap targets — great for efficiency, but it can cause self-reinforcing feedback loops (the "deadly triad" of function approximation + bootstrapping + off-policy).

W5. Why is a squared TD error a reasonable loss to minimise?

Q

satisfied Bellman, the two sides would be equal so the TD error is zero; squaring makes zero the unique minimum and penalises large violations more, giving a smooth gradient toward the fixed point.

W6. Why can't tabular Q-Learning scale to Atari pixels?

The state space (

210 \times 160 \times 3

images) is astronomically large and rarely revisited, so a table has no entry to look up and no way to generalise between similar screens.

Edge cases

EC1. What is

y

when the reward is

0

at a non-terminal step?

y = 0 + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})

— the future value still matters; a zero immediate reward does not zero the target.

EC2. What happens if every action has identical Q-value in a state under

ϵ

-greedy?

The

ar g max

ties, so greedy picks arbitrarily; exploration still happens via

ϵ

, so learning continues and the tie usually breaks as estimates diverge.

EC3. In the very first update, before any learning, are the targets meaningful?

Barely — with random

θ^{-}

the bootstrap is noise, but ==the immediate reward

r

y

is a real signal==, so learning slowly grounds the estimates from rewards outward.

EC4. What if the replay buffer only ever contains transitions with reward

0

(rewards are sparse)?

Q-values can stay flat until a rewarding transition is stored; this is the sparse-reward problem, motivating better exploration or reward shaping — replay reuses the rare rewarding sample many times once found.

EC5. A transition is stored, then the network improves a lot before it's sampled. Is the stored

(s, a, r, s^{'})

still valid?

Yes. The tuple records environment dynamics, not policy quality; the target

y

is recomputed with current

θ^{-}

at sample time, so stale data still yields a correct Bellman target.

EC6. What if

C = 1

(copy the target net every step)?

Then

θ^{-} \approx θ

always, so you have effectively no target network, reintroducing the moving-target instability the trick was meant to remove.

EC7. The same

(s, a)

leads to terminal reward

+ 10

half the time and

0

the other half. What does

Q (s, a)

learn, and why?

It learns the ==expected reward

5

==. The squared loss

E [(y - Q)^{2}]

is minimised where its derivative

- 2 E [y - Q] = 0

, i.e.

Q = E [y] = 0.5 (10) + 0.5 (0) = 5

. So a stochastic target pulls

Q

toward its mean, not toward either individual value.

Recall One-line summary of the traps

Detach the target, drop $max$ at terminals, DQN is off-policy with discrete actions, anneal $ϵ$ down, and never confuse "loss is zero" with "policy is optimal".

Connections

Deep Q-Networks (DQN) — parent topic this bank drills.
Double DQN — the fix for the overestimation trap (SE4).
Dueling DQN — alternative architecture, another common confusion.
Temporal Difference Learning — the bootstrapping idea behind W1/W2.
Epsilon-Greedy Exploration — source of the annealing trap (SE6).