Soft Actor-Critic (SAC) — Hinglish notes

5.2.11 · AI-ML › Deep & Advanced RL

SAC kyun exist karta hai?

Classic policy-gradient methods (jaise DDPG, TD3) mein do chronic pains hain:

Brittle exploration — ek deterministic policy bahut jaldi ek hi action par collapse ho jaati hai aur explore karna band kar deti hai.
Sample inefficiency vs stability trade-off — on-policy methods (PPO) stable hote hain par data waste karte hain; off-policy methods (DDPG) data reuse karte hain par unstable hote hain.

Objective kya hai? (Maximum-entropy RL)

Kyunki $- lo g π (a ∣ s)$ appear hota hai, hum entropy ko reward ke andar fold kar sakte hain ek soft reward $r + α H$ define karke. Baaki sab is augmented reward ke saath Bellman equations re-derive karke follow hota hai.

HOW: scratch se soft Bellman equations derive karna

Step 1 — Soft value. Soft state value define karo as expected return-plus-entropy from $s$ : $V (s) = E_{a \sim π} [Q (s, a) - α lo g π (a ∣ s)]$ Yeh step kyun? Extra $- α lo g π$ term exactly entropy contribution hai ( $H = E [- lo g π]$ ) jo $V$ se bahar nikala gaya hai. Toh value = expected Q plus policy kitni random hai.

Step 2 — Soft Q Bellman backup. Q-value = immediate reward + discounted next soft value: $Q (s, a) = r (s, a) + γ E_{s^{'}} [V (s^{'})]$ Yeh step kyun? Standard Bellman recursion, lekin $V$ already saare future steps ka entropy bonus carry karta hai. Toh future exploration bake in hai.

Step 3 — Optimal policy kya hai? Entropy term ke saath $max_{π} J$ solve karo. $E_{a} [Q - α lo g π]$ ka maximizer subject to $π$ being a distribution Boltzmann (softmax) policy hai: $π^{*} (a ∣ s) = \frac{e x p ( \frac{1}{α} Q ( s , a ) )}{Z ( s )}, Z (s) = \int exp (\frac{1}{α} Q (s, a^{'})) d a^{'}$

Teen learned pieces (training)

SAC seekhta hai: do Q-networks $Q_{θ_{1}}, Q_{θ_{2}}$ (+ unke targets), aur ek stochastic policy $π_{ϕ}$ .

Worked example 1 — soft target $y$ compute karna

Maano $r = 1$ , $γ = 0.99$ , next action $a^{'}$ deta hai $Q_{\overset{ˉ}{θ}_{1}} = 5.0$ , $Q_{\overset{ˉ}{θ}_{2}} = 4.6$ , $lo g π (a^{'} ∣ s^{'}) = - 1.2$ , $α = 0.2$ .

$min_{j} Q = 4.6$ . Kyun? Pessimistic estimate overestimation se bachata hai.
Entropy bonus $= - α lo g π = - 0.2 \times (- 1.2) = + 0.24$ . Kyun? Zyada random policy → higher next value.
$V (s^{'}) = 4.6 + 0.24 = 4.84$ .
$y = 1 + 0.99 \times 4.84 = 1 + 4.7916 = 5.79$ .

Worked example 2 — temperature $α$ ka effect

Do actions $Q = (2.0, 1.0)$ ke saath. Boltzmann policy $π \propto e^{Q / α}$ .

$α = 1$ : weights $e^{2}, e^{1} = 7.39, 2.72$ → $π = (0.73, 0.27)$ . Kyun? Mild preference, phir bhi explore karta hai.
$α = 0.1$ : weights $e^{20}, e^{10}$ → $π \approx (0.99995, 0.00005)$ . Kyun? Chota $α$ ⇒ near-greedy.
$α = 10$ : weights $e^{0.2}, e^{0.1}$ → $π \approx (0.525, 0.475)$ . Kyun? Bada $α$ ⇒ near-uniform, max exploration.

Recall Feynman: ek 12-saal ke bacche ko explain karo

Socho ek robot candy tak pahunchna seekh raha hai. Agar woh sirf candy ki parwah kare, toh woh EK raasta dhundh leta hai aur hamesha wahi chalta hai — chahe kal wahan wall aa jaaye. SAC robot ko thoda bonus deta hai alag raaste try karne ke liye ("surprising raho!"). Toh woh kuch achhe options khule rakhta hai aur stuck nahi hota. Knob $α$ decide karta hai ki adventurous hone ka kitna bonus milega. Do "judges" (do Q-networks) har move score karte hain aur hum hamesha zyada strict judge par trust karte hain taaki robot overconfident na ho.

Forecast-then-Verify

Answers: (1) deterministic/greedy — reward-only RL. (2) $min$ se overestimation counter karne ke liye. (3) off-policy: yeh replay data reuse karta hai, isliye PPO ke comparison mein bahut kam environment samples chahiye.

Flashcards

SAC kya objective maximize karta hai?

Expected return plus policy entropy:

\sum_{t} E [r + α H (π (\cdot ∣ s_{t}))]

Temperature

α

kya hai?

Woh coefficient jo reward aur entropy (exploration) ke beech trade-off karta hai; chota

α

→ greedy, bada

α

→ uniform.

Optimal soft policy form derive karo.

E [Q] - α E [lo g π]

maximize karne par

π^{*} \propto exp (Q / α)

milta hai (Boltzmann).

Soft state value

V (s)

likho.

V (s) = E_{a \sim π} [Q (s, a) - α lo g π (a ∣ s)]

Soft Q target

y

likho.

y = r + γ (min_{j} Q_{\overset{ˉ}{θ}_{j}} (s^{'}, a^{'}) - α lo g π (a^{'} ∣ s^{'}))

a^{'} \sim π

Do Q-networks ka min kyun lete hain?

Overestimation bias reduce karne ke liye (clipped double-Q, TD3 ki tarah).

Actor loss mein reparameterization trick kyun?

Sampled action ke through backpropagate karne ke liye;

a = tanh (μ + σ ⊙ ϵ)

Tanh log-prob correction kya hai?

lo g π (a) = lo g μ (u) - \sum_{i} lo g (1 - tanh^{2} u_{i})

α

auto-tune kaise hota hai?

E [- α (lo g π + \overset{ˉ}{H})]

minimize karo target entropy

\overset{ˉ}{H} = - dim (A)

ke saath.

SAC on-policy hai ya off-policy?

Off-policy; replay buffer use karta hai, isliye sample-efficient hai.

SAC kaun-kaun se networks train karta hai?

Do Q-critics (+targets) aur ek stochastic actor; value net optional hai.

Actor loss expression?

J_{π} = E [α lo g π_{ϕ} (a ∣ s) - min_{j} Q_{θ_{j}} (s, a)]

Connections

Soft Q-Learning — same $exp (Q / α)$ policy wala value-based ancestor.
TD3 — twin-Q / clipped double-Q trick ka source.
DDPG — deterministic off-policy actor-critic jise SAC improve karta hai.
Maximum Entropy RL — objective ke peeche ka framework.
Reparameterization Trick — low-variance stochastic actor gradients.
Bellman Equation — soft version yahan derive kiya gaya.
PPO — on-policy contrast; zyada stable par kam sample-efficient.
Entropy (Information Theory) — randomness ka $- E [lo g π]$ measure.