Reinforcement learning — MDP, Bellman equation, Q-learning

5.6.17 · Coding › Machine Learning (Aerospace Applications)

Intuition Badi picture (WHY yeh exist karta hai)

Supervised learning ko labeled answers chahiye. Lekin ek autopilot jo steering decide kar raha hai, uske paas koi teacher nahi hai jo har millisecond pe "correct" aileron angle chilla ke bata sake — usse sirf reward milta hai baad mein (smooth landing = achha, crash = bahut bura). Reinforcement Learning (RL) woh tarika hai jisse ek agent achha behavior sikhta hai purely trial, error, aur delayed reward se. Core problem yeh hai: abhi liya gaya ek action future mein bahut door tak rewards ko affect karta hai. Humein aise math ki zaroorat hai jo future value ko backwards propagate kare taaki present decisions guide ho sakein. Woh machinery hai MDP + Bellman equation, aur Q-learning ek practical algorithm hai jo use bina physics model jaane solve karta hai.

1. The Markov Decision Process (MDP)

2. Policies, Returns, aur Value

3. Bellman Equation Derive Karna (scratch se)

Ab policy $π$ ke under expectation lo: $V^{π} (s) = E_{π} [R_{t} + γ G_{t + 1} ∣ s_{t} = s]$ Expectation ko action ke upar, phir next state ke upar split karo: $V^{π} (s) = a \sum π (a ∣ s) s^{'} \sum P (s^{'} ∣ s, a) [R (s, a) + γ V^{π} (s^{'})]$

Figure — Reinforcement learning — MDP, Bellman equation, Q-learning

4. Q-learning ( $P$ jaane BINA solve karna)

5. Common Mistakes (Steel-manned)

6. Flashcards

Ek MDP ko define karne wale paanch elements kya hain?

States

S

, Actions

A

, Transition

P (s^{'} ∣ s, a)

, Reward

R

, Discount

γ

Markov property kya hai?

Next state sirf current state aur action pe depend karta hai, purani history pe nahi.

Hum future rewards ko

γ < 1

se discount kyun karte hain?

Infinite-horizon returns ko finite/convergent rakhne ke liye aur near-term rewards ko distant ones se zyada value dene ke liye.

Q^{*}

ke liye Bellman optimality equation batao.

Q^{*} (s, a) = \sum_{s^{'}} P (s^{'} ∣ s, a) [R (s, a) + γ max_{a^{'}} Q^{*} (s^{'}, a^{'})]

Q-learning update rule likho.

Q (s, a) \leftarrow Q (s, a) + α [r + γ max_{a^{'}} Q (s^{'}, a^{'}) - Q (s, a)]

TD error kya hai?

δ = r + γ max_{a^{'}} Q (s^{'}, a^{'}) - Q (s, a)

: Bellman target aur current estimate ke beech ka gap.

Q-learning ko "model-free" aur "off-policy" kyun kehte hain?

Model-free: yeh

P

use karne ki jagah real transitions sample karta hai. Off-policy:

max

target optimal policy seekhta hai jabki ek alag (

ε

-greedy) policy se explore karta hai.

Q-learning aur SARSA ke target mein kya difference hai?

Q-learning

max_{a^{'}} Q (s^{'}, a^{'})

use karta hai; SARSA

Q (s^{'}, a^{'})

use karta hai actually next liye gaye action ke liye.

Q^{*}

se optimal policy kaise recover karte hain?

π^{*} (s) = ar g max_{a} Q^{*} (s, a)

ε

-greedy kya balance karta hai?

Exploration (random actions) vs exploitation (greedy best-known action).

α = 0.5, γ = 0.9, Q = 2, r = 1, max Q^{'} = 6

diya ho, naya

Q

kya hoga?

target

= 1 + 0.9 \cdot 6 = 6.4

δ = 4.4

Q \leftarrow 2 + 0.5 \cdot 4.4 = 4.2

Recall Feynman: 12-saal ke bachche ko explain karo

Socho ek video game hai jahan tumhe points turant nahi milte — kabhi kabhi abhi liya achha move 10 moves baad payoff deta hai. Tum ek scorebook rakhte ho jo guess karta hai "agar main is situation mein yeh button press karun, toh total mein kitne points mil sakte hain?" Har baar jab tum khelto ho, tum dekhte ho tum kahan pahunche, wahan se kitne points mil sakte the, aur apna guess thoda zyada correct karte ho. Yeh hazaron baar karo aur tumhara scorebook tumhe har situation mein best button bata dega — chahe kisine tumhe game ke rules kabhi bataye hi nahi. Woh scorebook hai $Q$ , aur guess-fix-karna hai Q-learning.

Connections

Markov Chains — MDP = Markov chain + actions + rewards
Dynamic Programming — Bellman equation woh DP recursion hai; value/policy iteration use solve karte hain jab $P$ known ho
Stochastic Approximation — $α$ -step Q-update ko justify karta hai
Deep Q Networks (DQN) — bade state spaces ke liye $Q$ table ko neural net se replace karo
Optimal Control — Bellman ⟷ Hamilton–Jacobi–Bellman equation aerospace guidance mein
Gradient Descent — TD update squared TD error ka (semi-)gradient descent hai