Deep Q-Networks (DQN)

5.2.1 · AI-ML › Deep & Advanced RL

WHY karte hain hum DQN?

WHAT problem solve kar rahe hain? Hum chahte hain ek aisa agent jo actions choose kare taaki long-term reward maximise ho, aur environment ka bahut bada ya continuous state space ho (jaise raw Atari pixels: $210\times 160\times 3$ possible images — astronomically zyada states).

WHY nahi chalega plain Q-learning? Tabular Q-learning ek number $Q(s,a)$ store karta hai har (state, action) pair ke liye. Pixels ke liye aapko ek table chahiye jo universe ke atoms se bhi badi ho, aur aap kabhi same state dobara visit nahi karoge — koi generalisation nahi.

HOW DQN fix karta hai? Poori table ko approximate karo ek parametric function se: $Q(s,a;\theta )$ — ek neural net jiske weights $\theta$ hain. Nearby states weights share karte hain, isliye ek state ke baare mein seekhna generalise hota hai similar states ke liye.

Recap: kya hai $Q$? (first principles se derivation)

Bellman optimality equation derive karna. Sum ko split karo "pehla reward" + "baaki" mein:

$Q^{\ast }(s,a)=\mathbb{E}[r_t+\gamma {\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+1+k}]$

"Baaki" term, agar hum $s_{t+1}$ se optimally act karein, exactly ${\max }_{a^{\prime }}{Q}^{\ast }(s_{t+1},a^{\prime })$ hai. Toh:

Bellman ko loss mein badalna (DQN objective)

Hum chahte hain $Q(s,a;\theta )$ Bellman ko satisfy kare. Agar karta, toh dono sides equal hoti, isliye unka difference (the TD error) zero hota. Squared TD error minimize karo:

Target $y=r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }^{-})$ kyun hai aur true $Q^{\ast }$ kyun nahi? Hume $Q^{\ast }$ pata nahi! Isliye hum bootstrap karte hain: next state ki value ka current best estimate use karte hain ek stand-in ke roop mein. Yeh ek moving target hai — isliye exactly hume neeche dono tricks chahiye.

HOW update karte hain? Gradient descent on $\mathcal{L}$ ($y$ ko constant treat karte hue, matlab target ke through backprop mat karo): $\theta \leftarrow \theta +\alpha (y-Q(s,a;\theta )){\nabla }_{\theta }Q(s,a;\theta )$

Do tricks (yahi hai "Deep" ka secret sauce)

Poora DQN algorithm

1. Net $\theta$, target net ${\theta }^{-}=\theta$, aur empty replay buffer $D$ initialise karo.
2. Har step ke liye:
   • Action choose karo $ϵ$-greedy: prob $ϵ$ se random, warna $\arg {\max }_{a}Q(s,a;\theta )$.
   • $a$ execute karo, $r,s^{\prime }$ observe karo; $(s,a,r,s^{\prime })$ $D$ mein store karo.
   • $D$ se ek minibatch sample karo.
   • Targets compute karo $y=r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }^{-})$ ($y=r$ use karo agar $s^{\prime }$ terminal hai).
   • $(y-Q(s,a;\theta ){)}^2$ par gradient step lo.
   • Har $C$ steps mein: ${\theta }^{-}\leftarrow \theta$.

$ϵ$-greedy kyun? Pure greedy kabhi unknown actions explore nahi karta → stuck ho jaata hai. $ϵ$ exploration force karta hai; hum $ϵ$ ko ~1 se ~0.1 tak time ke saath anneal karte hain (pehle explore, baad mein exploit).

Worked Example 1 — haath se ek gradient target

State $s$, action $a$ liya, reward $r=1$ mila, $s^{\prime }$ mein land kiya (terminal nahi), $\gamma =0.9$. Target net kehta hai $Q(s^{\prime },\text{left})=2$, $Q(s^{\prime },\text{right})=5$. Current net: $Q(s,a)=3$.

• Step: ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })=\max (2,5)=5$. Kyun? Bellman best next action use karta hai.
• Step: $y=r+\gamma \cdot 5=1+0.9(5)=5.5$. Kyun? Yeh humara bootstrapped target hai.
• Step: TD error $=y-Q(s,a)=5.5-3=2.5$. Kyun? Net underestimate kar raha hai; $Q(s,a)$ ko $5.5$ ki taraf push karo.

Worked Example 2 — terminal state

Same lekin $s^{\prime }$ terminal hai (episode khatam hua). Tab $y=r=1$.

• Kyun? Termination ke baad koi future rewards nahi, isliye $\gamma \max Q$ term drop ho jaata hai.
• Agar purana $Q(s,a)=1.2$, TD error $=1-1.2=-0.2$ → $Q(s,a)$ ko thoda neeche nudge karo.

Flashcards

Connections

• Q-Learning — tabular ancestor jise DQN generalise karta hai.
• Bellman Equation — fixed point jo DQN ka loss target karta hai.
• Experience Replay — buffer trick, prioritized replay ka bhi base.
• Double DQN — DQN ke $\max$ overestimation bias ko fix karta hai.
• Dueling DQN — value aur advantage streams alag karta hai.
• Temporal Difference Learning — bootstrapping ka source.
• Policy Gradient Methods — continuous actions ke liye alternative family.
• Epsilon-Greedy Exploration — action selection strategy.

Concept Map