Soft Actor-Critic (SAC)

WHY does SAC exist?

Classic policy-gradient methods (like DDPG, TD3) have two chronic pains:

Brittle exploration — a deterministic policy collapses to one action too early and stops exploring.
Sample inefficiency vs stability trade-off — on-policy methods (PPO) are stable but throw away data; off-policy methods (DDPG) reuse data but are unstable.

WHAT is the objective? (Maximum-entropy RL)

Since $-\log\pi(a|s)$ appears, we can fold entropy into the reward by defining a soft reward $r + \alpha\mathcal{H}$ . Everything else follows by re-deriving Bellman equations with this augmented reward.

HOW: deriving the soft Bellman equations from scratch

Step 1 — Soft value. Define the soft state value as expected return-plus-entropy from $s$ : $V(s)=\mathbb{E}_{a\sim\pi}\big[Q(s,a)-\alpha\log\pi(a|s)\big]$ Why this step? The extra $-\alpha\log\pi$ term is exactly the entropy contribution ( $\mathcal{H}=\mathbb{E}[-\log\pi]$ ) pulled out of $V$ . So value = expected Q plus how random the policy is.

Step 2 — Soft Q Bellman backup. The Q-value = immediate reward + discounted next soft value: $Q(s,a)=r(s,a)+\gamma\,\mathbb{E}_{s'}\big[V(s')\big]$ Why this step? Standard Bellman recursion, but $V$ already carries the entropy bonus of all future steps. So future exploration is baked in.

Step 3 — What policy is optimal? Solve $\max_\pi J$ with the entropy term. The maximizer of $\mathbb{E}_a[Q-\alpha\log\pi]$ subject to $\pi$ being a distribution is the Boltzmann (softmax) policy: $\pi^*(a|s)=\frac{\exp\!\big(\tfrac{1}{\alpha}Q(s,a)\big)}{Z(s)},\qquad Z(s)=\int \exp\!\big(\tfrac{1}{\alpha}Q(s,a')\big)\,da'$

Formula Deriving the optimal soft policy (Lagrangian)

Maximize $\mathbb{E}_{a\sim\pi}[Q(s,a)]+\alpha\mathcal{H}(\pi)$ subject to $\int\pi\,da=1$ . Lagrangian: $L=\int\pi(a)\big(Q-\alpha\log\pi\big)da+\lambda\big(\int\pi\,da-1\big)$ . $\dfrac{\partial L}{\partial\pi}=Q-\alpha\log\pi-\alpha+\lambda=0 \Rightarrow \log\pi=\dfrac{Q}{\alpha}+\text{const}$ . Exponentiate and normalize $\Rightarrow \pi^*\propto\exp(Q/\alpha)$ . ✔ As $\alpha\to0$ this becomes greedy (deterministic); as $\alpha\to\infty$ it becomes uniform (pure exploration). This is why $\alpha$ is the exploration knob.

The three learned pieces (training)

SAC learns: two Q-networks $Q_{\theta_1},Q_{\theta_2}$ (+ their targets), and one stochastic policy $\pi_\phi$ .

Formula Critic loss (soft Q)

We regress each $Q_{\theta_i}$ toward the soft Bellman target using a replay buffer $\mathcal{D}$ : $J_Q(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim\mathcal{D}}\Big[\tfrac12\big(Q_{\theta_i}(s,a)-y\big)^2\Big]$ $y=r+\gamma\Big(\min_{j=1,2}Q_{\bar\theta_j}(s',a')-\alpha\log\pi_\phi(a'|s')\Big),\quad a'\sim\pi_\phi(\cdot|s')$ Why $\min$ of two Q's? Like TD3, taking the minimum fights overestimation bias — a single Q tends to over-report values because the max operator picks noisy positives. Why $-\alpha\log\pi$ inside the target? That is the $V(s')$ from Step 1: next-state value = expected Q minus log-prob (entropy bonus).

Worked example 1 — computing a soft target $y$

Suppose $r=1$ , $\gamma=0.99$ , next action $a'$ gives $Q_{\bar\theta_1}=5.0$ , $Q_{\bar\theta_2}=4.6$ , $\log\pi(a'|s')=-1.2$ , $\alpha=0.2$ .

$\min_j Q = 4.6$ . Why? Pessimistic estimate avoids overestimation.
Entropy bonus $=-\alpha\log\pi = -0.2\times(-1.2)=+0.24$ . Why? More random policy → higher next value.
$V(s') = 4.6 + 0.24 = 4.84$ .
$y = 1 + 0.99\times4.84 = 1 + 4.7916 = \mathbf{5.79}$ .

Worked example 2 — effect of temperature $\alpha$

Two actions with $Q=(2.0, 1.0)$ . Boltzmann policy $\pi\propto e^{Q/\alpha}$ .

$\alpha=1$ : weights $e^{2},e^{1}=7.39,2.72$ → $\pi=(0.73,0.27)$ . Why? Mild preference, still explores.
$\alpha=0.1$ : weights $e^{20},e^{10}$ → $\pi\approx(0.99995,0.00005)$ . Why? Small $\alpha$ ⇒ near-greedy.
$\alpha=10$ : weights $e^{0.2},e^{0.1}$ → $\pi\approx(0.525,0.475)$ . Why? Big $\alpha$ ⇒ near-uniform, max exploration.

Recall Feynman: explain to a 12-year-old

Imagine a robot learning to reach candy. If it only cares about candy, it finds ONE path and always walks it — even if a wall appears there tomorrow. SAC pays the robot a little bonus for trying different paths ("stay surprising!"). So it keeps a few good options open and doesn't get stuck. The knob $\alpha$ decides how much bonus it gets for being adventurous. Two "judges" (the two Q-networks) score each move and we always trust the stricter judge so the robot doesn't get overconfident.

Forecast-then-Verify

Answers: (1) deterministic/greedy — reward-only RL. (2) to counter overestimation via $\min$ . (3) off-policy: it reuses replay data, so it needs far fewer environment samples than PPO.

Flashcards

What objective does SAC maximize?

Expected return plus policy entropy:

\sum_t \mathbb{E}[r + \alpha\mathcal{H}(\pi(\cdot|s_t))]

What is the temperature

\alpha

The coefficient trading off reward vs entropy (exploration); small

\alpha

→ greedy, large

\alpha

→ uniform.

Derive the optimal soft policy form.

Maximizing

\mathbb{E}[Q]-\alpha\mathbb{E}[\log\pi]

gives

\pi^*\propto\exp(Q/\alpha)

(Boltzmann).

Write the soft state value

V(s)

V(s)=\mathbb{E}_{a\sim\pi}[Q(s,a)-\alpha\log\pi(a|s)]

Write the soft Q target

y

y=r+\gamma(\min_j Q_{\bar\theta_j}(s',a')-\alpha\log\pi(a'|s'))

a'\sim\pi

Why take the min of two Q-networks?

To reduce overestimation bias (clipped double-Q, like TD3).

Why the reparameterization trick in the actor loss?

To backpropagate through the sampled action;

a=\tanh(\mu+\sigma\odot\epsilon)

What is the tanh log-prob correction?

\log\pi(a)=\log\mu(u)-\sum_i\log(1-\tanh^2 u_i)

How is

\alpha

auto-tuned?

Minimize

\mathbb{E}[-\alpha(\log\pi+\bar{\mathcal H})]

with target entropy

\bar{\mathcal H}=-\dim(\mathcal A)

Is SAC on- or off-policy?

Off-policy; uses a replay buffer, making it sample-efficient.

What networks does SAC train?

Two Q-critics (+targets) and one stochastic actor; value net is optional.

Actor loss expression?

J_\pi=\mathbb{E}[\alpha\log\pi_\phi(a|s)-\min_j Q_{\theta_j}(s,a)]

Connections

Soft Q-Learning — the value-based ancestor with the same $\exp(Q/\alpha)$ policy.
TD3 — source of the twin-Q / clipped double-Q trick.
DDPG — deterministic off-policy actor-critic SAC improves on.
Maximum Entropy RL — the framework behind the objective.
Reparameterization Trick — low-variance stochastic actor gradients.
Bellman Equation — soft version derived here.
PPO — on-policy contrast; more stable but less sample-efficient.
Entropy (Information Theory) — the $-\mathbb{E}[\log\pi]$ measure of randomness.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

SAC ka core idea simple hai: normal RL sirf reward maximize karta hai, lekin SAC reward ke saath-saath policy ki entropy (yaani randomness) ko bhi maximize karta hai. Iska matlab agent ko fayda milta hai jab woh success bhi paaye aur thoda "unpredictable" bhi rahe. Isse exploration apne aap zinda rehti hai — agent ek hi action pe stuck nahi hota, kyunki uspe "adventurous raho" ka bonus milta hai. Yeh bonus kitna strong ho, woh temperature $\alpha$ decide karta hai: chhota $\alpha$ matlab greedy (ek hi best action), bada $\alpha$ matlab sab actions barabar (full exploration).

Maths side pe, hum normal Bellman equation ko "soft" bana dete hain. Value $V(s)=\mathbb{E}[Q-\alpha\log\pi]$ — yani expected Q plus entropy ka bonus. Jab hum entropy ke saath objective ko maximize karte hain, toh optimal policy $\pi^*\propto \exp(Q/\alpha)$ nikalti hai — yeh ek softmax/Boltzmann distribution hai. Yehi $\exp(Q/\alpha)$ formula batata hai ki kyun $\alpha$ exploration ka knob hai.

Practical training mein SAC teen cheezein seekhta hai: do Q-networks (twin critics), aur ek stochastic actor. Do Q-networks ka $\min$ lete hain taaki Q overestimate na ho (TD3 wala trick). Actor ko update karte hain reparameterization trick se — $a=\tanh(\mu+\sigma\epsilon)$ — taaki gradient sample ke through flow kare. Ek important detail: tanh lagane ke baad log-probability mein $\log(1-\tanh^2)$ correction add karna padta hai, warna entropy galat aayegi.

Yeh matter kyun karta hai? SAC off-policy hai, matlab replay buffer se purana data reuse karta hai — isliye bahut sample-efficient hai, PPO se kam environment interactions chahiye. Aur entropy ki wajah se training stable aur robust rehti hai. Continuous control (robotics, locomotion) mein SAC aaj bhi ek top baseline hai — yaad rakho: Soft targets, Adventurous actor, Cautious critic.

WHY does SAC exist?

WHAT is the objective? (Maximum-entropy RL)

HOW: deriving the soft Bellman equations from scratch

The three learned pieces (training)

Worked example 1 — computing a soft target $y$

Worked example 2 — effect of temperature $\alpha$

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Deep & Advanced RL

Connections