World models and embodied AI

6.5.6 · AI-ML › Research Frontiers & Practice

What Are World Models?

Yeh kyun chahiye?

• Sample efficiency: Real-world interactions mehengi hoti hain (robot toot sakta hai, medical trial mein jaan ja sakti hai). Apne dimaag mein (model mein) rollouts simulate karo.
• Planning: Model ke saath, aap act karne se pehle future trajectories simulate karke "aage soch" sakte ho (jaise chess players moves visualize karte hain).
• Transfer: Ek accha world model environment dynamics capture karta hai jo tasks mein transfer hoti hain (same physics, alag goals).

Yeh model-free RL se alag kaise hai?

• Model-free (DQN, PPO): Trial-and-error se seedha $Q(s,a)$ ya $\pi (a|s)$ seekhta hai. Consequences ki koi explicit samajh nahi.
• Model-based: Pehle $p(s_{t+1}|s_t,a_t)$ seekhta hai, phir policy plan karne ya train karne ke liye use karta hai. Samajhta hai "agar main yeh push karunga, toh yeh giregi."

Derivation: How World Models Enable Planning

Step 1: The Forward Model

Markov Decision Process (MDP) dynamics se shuru karo: $s_{t+1}\sim P(s_{t+1}|s_t,a_t)$

Model-free RL mein, hum $P$ explicitly kabhi nahi seekhte. Hum sirf samples se value function $V(s)$ ya policy $\pi (a|s)$ seekhte hain.

World model approach: Ek approximation ${\hat{P}}_{\theta }(s_{t+1}|s_t,a_t)$ seekho (yeh forward model hai).

Yeh explicitly kyun seekhein? Kyunki ek baar yeh mil jaaye, toh hum kar sakte hain:

1. Real environment interaction ke bina hypothetical trajectories rollout karna
2. Actions optimize karne ke liye model ke through backpropagate karna
3. Data augmentation synthetic experience simulate karke

Step 2: Planning with the Model

${\hat{P}}_{\theta }$ diye hue, hum best action sequence dhundhna chahte hain. Model Predictive Control (MPC) use karo:

$a_{0:H-1}^{\ast }=\arg {\max }_{a_{0:H-1}}{\mathbb{E}}_{{\hat{P}}_{\theta }}\left[{\sum }_{t=0}^{H-1}r(s_t,a_t)\right]$

First principles se derivation:

Trajectory return se shuru karo: $J(\tau )={\sum }_{t=0}^{H-1}r(s_t,a_t)$

Hum apni learned dynamics ke under expected return maximize karna chahte hain: $J(a_{0:H-1})={\mathbb{E}}_{s_1\sim \hat{P}(\cdot |s_0,a_0)}{\mathbb{E}}_{s_2\sim \hat{P}(\cdot |s_1,a_1)}\cdots \left[{\sum }_{t=0}^{H-1}r(s_t,a_t)\right]$

Yeh step kyun? Har future state $s_t$ random hai kyunki model stochastic hai. Hum saari possible futures par integrate karte hain unki probability se weighted.

Nested expectation expand karke: $J=\int \hat{P}(s_1|s_0,a_0)\left[r(s_0,a_0)+\int \hat{P}(s_2|s_1,a_1)\left[r(s_1,a_1)+\cdots \right]d{s}_2\right]d{s}_1$

Recursive kyun? Return time ke saath additive hai, aur har state sirf previous par depend karta hai (Markov property).

Practice mein, hum is integral ko is tarah approximate karte hain:

• ${\hat{P}}_{\theta }$ se $K$ trajectories sample karo
• Unke returns average karo
• Gradient descent ya shooting methods (CEM, CMA-ES) se $a_{0:H-1}$ optimize karo

$J\approx \frac{1}{K}{\sum }_{k=1}^K{\sum }_{t=0}^{H-1}r(s_t^{(k)},a_t)\text{where }{s}_t^{(k)}\sim {\hat{P}}_{\theta }(\cdot |s_{t-1}^{(k)},a_{t-1})$

Key insight: Model humein action sequences reality mein commit karne se pehle simulation mein evaluate karne deta hai.

Step 3: Dyna Architecture (Learning + Planning)

Dyna framework model-learning aur model-free RL ko unify karta hai:

1. Real experience: $(s,a,r,s^{\prime })\to$ policy $\pi$ update karo (model-free) AUR model ${\hat{P}}_{\theta }$ update karo
2. Simulated experience: $(s,a)\sim$ replay buffer sample karo, $s^{\prime }\sim {\hat{P}}_{\theta }(s^{\prime }|s,a)$ generate karo → policy update karo

Yeh kyun kaam karta hai?

• Real data scarce hai → model use karo synthetic data generate karne ke liye
• Model errors long rollouts mein compound hoti hain → short $k$-step rollouts use karo
• Policy faster improve hoti hai kyunki use real sample per $N_{\text{real}}+N_{\text{sim}}$ updates milte hain

Dyna update rule: ${\theta }_{\pi }\leftarrow {\theta }_{\pi }+\alpha \left[{\nabla }_{\theta }{J}_{\text{real}}+\lambda {\nabla }_{\theta }{J}_{\text{sim}}\right]$ jahan $\lambda$ model trust ko trade off karta hai.

$\lambda$ weight kyun? Agar model poor hai, toh simulated gradients biased hain → unka influence kam karo.

What Is Embodied AI?

Embodiment kyun matter karta hai:

• Grounding: Abstract concepts (jaise "bhaari") physical interaction ke zariye seekhe jaate hain, definitions se nahi
• Affordances: Aap kya kar sakte ho yeh aapke body par depend karta hai (ek quadruped door us tarah nahi khol sakta jaise ek humanoid)
• Exploration: Embodied agents naturally move, manipulate, try karke explore karte hain—yeh diverse data generate karta hai
• Partial observability: Real bodies mein limited sensors hote hain → agent ko actively information gather karni padti hai

Historical context: Classical AI (GOFAI) ne logic aur symbols ke zariye intelligence build karne ki koshish ki. Embodied cognition movement ne argue kiya ki yeh ignore karta hai ki biological intelligence body se inseparable hai.

Architecture: World Models (Ha & Schmidhuber, 2018)

Seminal World Models paper ne ek three-component architecture propose ki:

1. V (Vision): Variational autoencoder observation $o_t$ ka latent representation $z_t$ seekhta hai $z_t=\text{Encoder}(o_t),o_t\approx \text{Decoder}(z_t)$

2. M (Memory): Mixture Density Network RNN next latent state predict karta hai $P(z_{t+1}|z_t,a_t,h_t)={\sum }_{i=1}^K{\pi }_i\mathcal{N}({\mu }_i,{\sigma }_i^2)$ jahan $h_t$ RNN hidden state hai.

3. C (Controller): Small linear policy $a_t=W{z}_t+Wh{h}_t+b$

Yeh architecture kyun?

• V high-dim observations (pixels) ko compress karta hai → low-dim $z_t$ mein (jaise "aage ek car hai")
• M temporal dynamics seekhta hai → future $z_{t+1}$ predict karta hai (jaise "agar main left steer karunga, toh car mere view mein right move karegi")
• C tiny hai kyunki yeh abstract $z_t$ par operate karta hai, raw pixels par nahi → optimize karna aasaan hai

Key innovation: M ko real trajectories par train karo, phir C ko entirely M ke dream mein train karo (simulated rollouts). Initial data collect karne ke baad koi real environment nahi chahiye.

M ke loss ki derivation:

Hum $P(z_{t+1}|z_t,a_t)$ chahte hain. Maximum likelihood use karo: ${\mathcal{L}}_M=-\log P_M(z_{t+1}|z_t,a_t,h_t)$

Mixture Density Network ke liye: $P_M(z_{t+1})={\sum }_{i=1}^K{\pi }_i(z_t,a_t,h_t)\cdot \mathcal{N}(z_{t+1};{\mu }_i,{\sigma }_i^2)$

Gaussians ka mixture kyun? Kyunki future stochastic aur multimodal hai (jaise ek car left OR right turn kar sakti hai). Ek single Gaussian average predict karta (seedha jaana), jo galat hai.

Negative log-likelihood: ${\mathcal{L}}_M=-\log \left({\sum }_{i=1}^K{\pi }_i\exp \left(-\frac{(z_{t+1}-{\mu }_i{)}^2}{2{\sigma }_i^2}\right)\right)$

Yeh loss kyun? Yeh true next state distribution aur hamare model ki prediction ke beech cross-entropy hai. Ise minimize karne se model reality se match karta hai.

Common Mistakes

Connections to Other Concepts

• Model-FreeRL: World models complement hain—model-based methods ko sample efficiency chahiye lekin train karna complex hai
• Variational-Autoencoders: VAEs latent state representation ke liye V (Vision) component provide karte hain
• Recurrent-Neural-Networks: RNNs (LSTMs, GRUs) temporal dynamics ke liye standard M (Memory) hain
• Reinforcement-Learning-Basics: World models MDP framework mein fit hote hain, transition function $P(s^{\prime }|s,a)$ seekhte hain
• Imitation-Learning: Behavioral cloning world models ignore karta hai; inverse RL reward seekh sakta hai, phir plan karne ke liye model use kar sakta hai
• Sim-to-Real-Transfer: Embodied AI ko simulated world models aur real physics ke beech "reality gap" bridge karna padta hai
• Vision-Transformers: Modern V components better representation ke liye CNs/VAEs ki jagah ViT encoders use kar sakte hain
• Diffusion-Models: Generative world models ke liye emerging approach (future frames ko denoising process ke roop mein predict karna)

#flashcards/ai-ml

Reinforcement learning mein world model kya hota hai? :: Ek learned probabilistic model $p(s_{t+1},r_t|s_t,a_t)$ jo current state aur action diye hue next state aur reward predict karta hai—environment dynamics ka internal simulation.

Concept Map