Experience replay — Hinglish notes

5.2.2 · AI-ML › Deep & Advanced RL

Experience Replay KYA hai?

Is trick ka typical ghar Deep Q-Networks (DQN) hai, isliye hum sab kuch wahan ground karenge.

Hume iska kyun zaroorat hai? (woh do diseases jinhe ye theek karta hai)

Naive "online" deep RL network ko har transition se update karta hai jab woh aata hai. Do problems aati hain:

Disease 1 — Correlated samples. Consecutive states $s_{t}, s_{t + 1}, s_{t + 2}$ almost identical hote hain (car 3 pixels move ki). Stochastic gradient descent assume karta hai ki aapke samples roughly i.i.d. hain. Usse near-duplicate, highly-correlated samples ki stream dene se high-variance, biased gradient estimates milti hain — net jo abhi dekh raha hai uspar over-fit ho jaata hai.

Disease 2 — Data inefficiency. Ek rare, informative experience (aakhirkar ek point score kiya!) ek baar dekha jaata hai aur discard ho jaata hai. Buffer ke saath isse dozens of times replay kiya ja sakta hai.

Disease 3 — Non-stationary target chasing. Policy badal jaati hai → aane wale data ki distribution badal jaati hai → network ek moving target chase karta hai aur oscillate/diverge kar sakta hai. Ek bade buffer par average karna training distribution ko smooth karta hai.

Ye kaise kaam karta hai? (algorithm, step by step derive kiya hua)

Hum $Q_{θ} (s, a)$ ko Bellman optimality equation satisfy karne ke liye fit karna chahte hain: $Q^{*} (s, a) = E_{s^{'}} [r + γ max_{a^{'}} Q^{*} (s^{'}, a^{'})] .$

Kyunki hume $Q^{*}$ nahi pata, hum bootstrap karte hain: current network use karke ek target banate hain, $y = r + γ max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'}),$ jahan $θ^{-}$ ek (slowly updated) target-network copy hai. Ek transition ke liye loss squared TD error hai: $ℓ (e; θ) = (y - Q_{θ} (s, a))^{2} .$

Yahaan buffer se sample kyun karein? True objective state–action distribution $ρ$ par expectation hai: $L (θ) = E_{e \sim ρ} [ℓ (e; θ)] .$ $\nabla_{θ} L$ ka unbiased Monte-Carlo estimate chahiye toh samples $ρ$ se independently drawn hone chahiye. Ek bade buffer se uniform sampling, correlated live stream se kahin behtar i.i.d. draws approximate karta hai. Isliye: $\nabla_{θ} L (θ) \approx \frac{1}{∣ B ∣} \sum_{e \in B} \nabla_{θ} ℓ (e; θ), B \sim U (D) .$

Prioritized Experience Replay (80/20 upgrade)

Uniform sampling un transitions par effort waste karta hai jinhein net already achhi tarah predict karta hai. Prioritized Experience Replay (PER) TD error $δ_{i} = y_{i} - Q_{θ} (s_{i}, a_{i})$ ke magnitude ke proportion mein sample karta hai — "woh transitions jinpar main sabse zyada galat hoon, unhe hi mujhe study karna chahiye."

Worked examples

Common mistakes (steel-manned)

Forecast-then-Verify

Recall Answer padhne se pehle forecast karo

Q: Agar aap PER mein $α = 0$ set karo, toh kaunsa algorithm recover hota hai, aur kyun? A: Plain (uniform) experience replay — har $p_{i}^{0} = 1$ , isliye sabke liye $P (i) = 1/ N$ . Q: Buffer size 100× badhao; do randomly sampled transitions ke beech correlation upar jaayegi ya neeche? A: Neeche — bada, zyada diverse pool ⇒ do random picks ke temporally adjacent hone ki probability kam ⇒ zyada i.i.d.-jaisa.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tum ek video game seekh rahe ho. Ek silly tarika hai sirf woh move sochna jo abhi abhi ki aur phir usse hamesha ke liye bhool jaana. Ek smart tarika hai: purani moves aur jo hua uska ek notebook rakhna. Har practice round mein kuch random pages paltao aur review karo. Random pages tumhe sirf aakhri cheez ke baare mein sochte rehne se rokti hain, aur purane pages review karne ka matlab hai ek baar mila lucky trick nahi bhoolta. Agar kuch pages par "yahan main bahut galat tha" mark ho, toh tum woh pages zyada baar paltate ho — lekin apne aap ko yaad dilaate ho ki ye rare hain taaki tumhe apni ability ki galat picture na mile.

Flashcards

Online deep RL mein experience replay pehli kaunsi problem solve karta hai?

Yeh consecutive (near-identical) samples ke beech correlation todhta hai, SGD ke liye ~i.i.d. assumption restore karta hai.

Replay buffer mein kya store hota hai?

Transitions

(s_{t}, a_{t}, r_{t}, s_{t + 1})

(plus ek terminal flag), ek fixed-capacity FIFO mein.

Plain experience replay mein mini-batches kaise draw kiye jaate hain?

Buffer se uniformly at random.

Replay data efficiency kyun improve karta hai?

Har transition (especially rare informative wale) ko ek baar ki jagah kaafi saare gradient updates mein reuse kiya ja sakta hai.

Replay sirf off-policy algorithms ke liye hi valid kyun hai?

Buffer mein purani policies ka data hota hai; on-policy methods ko current policy ke samples chahiye.

DQN with replay mein TD target kya hai?

y = r + γ (1 - d) max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'})

PER mein kisi transition ki priority kya determine karta hai?

Uske TD error ka magnitude,

p_{i} = ∣ δ_{i} ∣ + ϵ

PER sampling probability do.

P (i) = p_{i}^{α} / \sum_{k} p_{k}^{α}

PER ko importance-sampling weights kyun chahiye?

Non-uniform sampling gradient estimate ko bias karta hai;

w_{i} = (N P (i))^{- β}

us bias ko correct karta hai.

PER mein

α = 0

kya deta hai?

Uniform experience replay (sab priorities equal).

Replay ke saath target network

θ^{-}

kya role play karta hai?

Ek slowly-changing target provide karta hai taaki net ek moving objective chase na kare, stability add karta hai.

Connections

Deep Q-Networks (DQN) — woh algorithm jiske liye replay design kiya gaya tha
Bellman optimality equation — TD target ka source
Temporal-Difference Learning — woh update jise replay feed karta hai
Off-policy vs On-policy — replay ko off-policy learning kyun chahiye
Importance Sampling — PER mein use hone wala bias correction
Target Networks — companion stabiliser
DDPG / Soft Actor-Critic — off-policy actor-critics jo replay bhi use karte hain