Model-based RL overview — Hinglish notes

5.2.14 · AI-ML › Deep & Advanced RL

Intuition Bada picture

RL mein, ek agent trial and error se act karna seekhta hai. Do grand strategies hain:

Model-free RL: experience se directly policy/value seekho. "Bas cheezein try karte raho jab tak acche na ho jao."
Model-based RL (MBRL): pehle environment ka model of the environment seekho (ya use karo) — ki world aapke actions pe kaise respond karti hai — aur phir us model ke andar plan karo.

Core WHY: ek model aapko consequences imagine karne deta hai bina real-world cost chukaye. Real interaction expensive hoti hai (ek robot arm toot sakta hai, ek real trade mein paisa jaata hai), lekin apne imagination mein imagined trajectories rollout karna sasta hai. Isliye MBRL zyada computation ko kam real data ke badle trade karta hai — yeh usually bahut zyada sample-efficient hota hai.

"The model" exactly kya hai?

WHAT model aapko deta hai: ek sasta simulator. WHY hum ise chahte hain: plan karne ke liye / synthetic experience generate karne ke liye. HOW hum ise paate hain: ya toh yeh given hai (chess ke rules exactly known hain) ya phir supervised regression se data se seekha jaata hai.

Model use karne ke do flavours

Learned model ko regression se derive karna (scratch se)

Hum koi formula seedha nahi denge. Isko build karte hain.

Step 1 — Hum kya observe karte hain? Data tuples $D = {(s_{i}, a_{i}, s_{i}^{'}, r_{i})}$ acting se collect kiye. Yeh step kyun? Kyunki ek model ko real cause→effect samples pe train karna padta hai.

Step 2 — Target choose karo. Hum ek function $\hat{f}_{θ} (s, a)$ chahte hain jo $s^{'}$ predict kare. Natural learning objective yeh hai ki predictions reality se match karein. Gaussian noise assumption $s^{'} = f (s, a) + ε$ , $ε \sim N (0, σ^{2} I)$ use karke, maximum likelihood least squares ban jaata hai. Yeh step kyun? Gaussian MLE ⇔ squared error minimize karo — loss yahan se aata hai, magic nahi hai.

Planning derive karna: kyun ek model aapko value deta hai

Jab aapke paas $\hat{P}, \overset{r}{^}$ ho, planning wohi Bellman idea use karti hai jaise model-free — lekin internally compute kiya jaata hai.

Continuous control ke liye hum aksar trajectory optimization / MPC use karte hain: current $s_{0}$ se, action sequence $a_{0 : H}$ search karo jo maximize kare $\sum_{t = 0}^{H - 1} γ^{t} \overset{r}{^} (\overset{s}{^}_{t}, a_{t}), \overset{s}{^}_{t + 1} = \hat{f} (\overset{s}{^}_{t}, a_{t}),$ sirf $a_{0}$ execute karo, phir re-plan karo. WHY har step pe re-plan? Compounding model error correct karne ke liye (neeche steel-man).

Canonical loop

Worked examples

Common mistakes (steel-manned)

80/20 — woh 20% jo 80% deta hai

Feynman

Recall Ek 12-saal ke bachche ko explain karo

Socho tum ek video game seekh rahe ho. Ek model-free bachcha bas buttons press karta rehta hai jab tak jeet na jaye — bahut time lagta hai aur kaafi lives jaati hain. Ek model-based bachcha pehle rules samajhta hai ("agar main yahan jump karun toh wahan land karunga"), phir level apni imagination mein plan test karne ke liye khelta hai, aur sirf tab buttons press karta hai jab soch liya. Woh bahut kam real lives kharach karta hai. Khatara yeh hai: agar unke imagined rules thodi galat hain, toh unka imagined plan bilkul galat ho sakta hai — isliye woh sirf kuch steps aage tak apni imagination pe trust karte hain, phir real screen dobara dekhte hain.

Flashcards

Ek environment model kya do cheezein predict karta hai?

Transition

\hat{P} (s^{'} ∣ s, a)

(next state) aur reward

\overset{r}{^} (s, a)

Model-based vs model-free RL ka core trade-off kya hai?

MBRL zyada computation use karta hai (planning/imagining) zyada sample efficiency achieve karne ke liye (kam real environment interactions).

Least-squares model-fitting loss kahan se aata hai?

Gaussian next-state noise assumption ke under maximum likelihood se; log-likelihood reduce hoti hai

- \frac{1}{2 σ ^{2}} ∥ s^{'} - \hat{f}_{θ} (s, a) ∥^{2}

mein.

s^{'}

directly predict karne ki jagah

Δ s = s^{'} - s

kyun predict karo?

Change chhota aur stable hai regress karne ke liye (states slowly change hoti hain), large-magnitude near-identity targets se bachte hain; baad mein

s

add karo.

Learned MBRL ka #1 failure mode kya hai?

Long rollouts mein compounding model error — small per-step errors accumulate hoti hain, isliye plans ek galat "fantasy" model ke against banaye jaate hain.

Compounding model error ke liye do remedies?

Frequent re-planning ke saath short planning horizons (MPC) aur model ensembles / uncertainty estimation.

Dyna ka key idea kya hai?

Learned model use karke synthetic (imagined) transitions generate karo aur unpe extra value/policy updates run karo, scarce real data ko amplify karke.

MPC mein, optimized sequence ka sirf pehla action execute kyun karte hain?

Har step pe true observed next state se re-plan karne ke liye, model inaccuracies correct karne ke liye.

"Model given" aur "model learned" settings mein kya fark hai?

Given = exact rules known hain (e.g. chess/AlphaZero), sirf planning hard hai; learned = data se dynamics fit karo (e.g. robotics), model-error problems add ho jaati hain.

Model ke andar planning ke liye use hone wali Bellman optimality equation?

V^{*} (s) = max_{a} [\overset{r}{^} (s, a) + γ \sum_{s^{'}} \hat{P} (s^{'} ∣ s, a) V^{*} (s^{'})]

Connections

Model-free RL — alternative family (policy/value directly seekho).
Dyna Architecture — learning, planning, aur acting ko integrate karna.
Model Predictive Control (MPC) — plan karo, pehla action execute karo, re-plan karo.
Bellman Equation — woh recursion jis par planning rely karti hai.
MuZero and Dreamer — value-aware / latent world models.
Sample Efficiency — woh metric jo MBRL optimize karta hai.
Markov Decision Process — woh formal object jo model kiya ja raha hai.
Uncertainty and Ensembles in RL — compounding error se ladna.