Exercises — Soft Actor-Critic (SAC)

SAC maximizes expected reward plus policy entropy. Entropy $\mathcal{H}(\pi (\cdot |s))=-{\mathbb{E}}_{a\sim \pi }[\log \pi (a|s)]$ measures how spread out (unpredictable) the action distribution is. The objective is $J(\pi )={\sum }_t{\mathbb{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}[r(s_t,a_t)+\alpha \mathcal{H}(\pi (\cdot \mid s_t)\left)\right].$ Here ${\rho }_{\pi }$ just means "states and actions visited when you follow policy $\pi$", and $\alpha >0$ is the temperature knob.

Answer: (b). The extra term $-\alpha \log \pi (a|s)$ is the entropy contribution, because $\mathcal{H}={\mathbb{E}}_a[-\log \pi ]$. Option (a) forgets exploration; option (c) is the hard (deterministic) value used by DDPG/TD3, not the soft one.

It combats overestimation bias — a single Q-network tends to over-report values because bootstrapping through a $\max$-like operator systematically picks up noisy positive errors. The clipped double-Q trick comes from TD3.

Step 1 — pick the pessimistic Q. $\min (3.0,3.4)=3.0$. Step 2 — entropy bonus. $-\alpha \log \pi =-0.5\times (-0.8)=+0.4$. Positive because a more random policy earns more future value. Step 3 — soft next value. $V(s^{\prime })=3.0+0.4=3.4$. Step 4 — Bellman backup. $y=0.5+0.98\times 3.4=0.5+3.332=3.832$.

Weights: $e^{3/1}=e^3=20.0855$, $e^{1/1}=e^1=2.71828$. Sum $=22.8038$. $\pi =\left(\frac{20.0855}{22.8038},\frac{2.71828}{22.8038}\right)=(0.8808,0.1192).$ A strong-but-not-total preference for the higher-Q action — exactly the "hedge" behaviour entropy buys us.

$-0.8808\ln (0.8808)=-0.8808\times (-0.12692)=0.11179$. $-0.1192\ln (0.1192)=-0.1192\times (-2.12730)=0.25357$. $\mathcal{H}=0.11179+0.25357=0.3654$ nats. For comparison, a uniform two-action policy has $\mathcal{H}=\ln 2=0.6931$, the maximum. Our policy is more decided, so its entropy is lower.

$\alpha =0.2$: exponents $3/0.2=15$, $1/0.2=5$. Weights $e^{15}=3269017$, $e^5=148.413$. Sum $=3269165.4$. $\pi =(0.99995460,0.00004540)\approx (1.0,0.0).$ Near-greedy: small $\alpha$ ⇒ the exponent gap explodes ⇒ deterministic. $\alpha =5$: exponents $3/5=0.6$, $1/5=0.2$. Weights $e^{0.6}=1.82212$, $e^{0.2}=1.22140$. Sum $=3.04352$. $\pi =(0.59869,0.40131)\approx (0.60,0.40).$ Near-uniform: large $\alpha$ flattens the difference ⇒ heavy exploration. Both limits match the "$\alpha$ is the exploration knob" claim.

With $\max (3.0,3.4)=3.4$: $V(s^{\prime })=3.4+0.4=3.8$, and $y=0.5+0.98\times 3.8=0.5+3.724=4.224$. Change: $4.224-3.832=+0.392$ — the target is inflated. Why dangerous: Q-networks are regressed toward these targets. If targets are systematically too high, the critic learns inflated values, the actor chases those illusory rewards, and the error compounds through bootstrapping. The $\min$ deliberately biases downward to cancel the upward noise — a controlled pessimism.

$\tanh (0.5)=0.46212$, so ${\tanh }^2(0.5)=0.21355$, and $1-{\tanh }^2=0.78645$. The term inside the subtraction is $\log (1-{\tanh }^2)=\log (0.78645)=-0.24011$. So we subtract a negative number: $\log \pi =\log \mu (u)-\log (1-{\tanh }^{2}u)=-1.0-(-0.24011)=-1.0+0.24011=-0.75989.$ Sign meaning: since $1-{\tanh }^2\le 1$, we always have $\log (1-{\tanh }^2)\le 0$. Subtracting a non-positive quantity adds to $\log \pi$, making it less negative — the transformed log-density is higher than the raw Gaussian log-density. Why? The squash $a=\tanh (u)$ compresses a wide range of $u$ into the narrow interval $(-1,1)$; probability mass gets concentrated, so density rises. Look at the figure: near $u=0$ the tanh is nearly linear so the correction is small; near the flat tails the squash crushes a wide $u$-range into a tiny $a$-range, concentrating density most.

Objective as a function of $p$: $f(p)=p{q}_1+(1-p)q_2-\alpha (p\ln p+(1-p)\ln (1-p)).$ (The bracket is $-\mathcal{H}$, so subtracting $\alpha$ times it adds $\alpha \mathcal{H}$.) Differentiate and set to zero: $f^{\prime }(p)=q_1-q_2-\alpha (\ln p+1-\ln (1-p)-1)=q_1-q_2-\alpha \ln \frac{p}{1-p}=0.$ Solve: $\ln \frac{p}{1-p}=\frac{q_1-q_2}{\alpha }\Rightarrow \frac{p}{1-p}=e^{(q_1-q_2)/\alpha }$, giving $p=\frac{e^{q_1/\alpha }}{e^{q_1/\alpha }+e^{q_2/\alpha }},$ which is exactly the Boltzmann policy ${\pi }^{\ast }\propto \exp (Q/\alpha )$. ✔ Plug in $q_1=3,q_2=1,\alpha =1$: $p=\frac{e^3}{e^3+e^1}=\frac{20.0855}{22.8038}=0.8808$ — matches Problem 2.2.

Note $\mathbb{E}[\log \pi ]=-{\mathcal{H}}_{\text{current}}$. So ${\nabla }_{\alpha }J=-(\mathbb{E}[\log \pi ]+\bar{\mathcal{H}})=-(-{\mathcal{H}}_{\text{current}}+\bar{\mathcal{H}})={\mathcal{H}}_{\text{current}}-\bar{\mathcal{H}}.$ Plug in: ${\nabla }_{\alpha }J=0.37-0.69=-0.32$. Gradient descent updates $\alpha \leftarrow \alpha -\eta {\nabla }_{\alpha }J=\alpha -\eta (-0.32)=\alpha +0.32\eta$, so $\alpha$ increases. Interpretation: current entropy ($0.37$) is below target ($0.69$) → the policy is too decisive → raise $\alpha$ to reward exploration more, pushing entropy back up. Self-annealing, exactly as advertised.

${\min }_{j}Q=\min (2.2,2.0)=2.0$. $\alpha \log \pi =0.5\times (-0.90)=-0.45$. Loss $=-0.45-2.0=-2.45$. Minimizing this loss pushes toward more negative values, i.e. higher $Q$ (the $-{\min }_{j}Q$ term) and higher entropy / lower $\log \pi$ (the $\alpha \log \pi$ term, since making $\log \pi$ more negative lowers the loss). Both goals — be good and stay surprising — pull in the same optimization.

From $p=\frac{1}{1+e^{-(q_1-q_2)/\alpha }}$. Let $\Delta =q_1-q_2>0$. As $\alpha \to 0^{+}$, $\frac{\Delta }{\alpha }\to +\infty$, so $e^{-\Delta /\alpha }\to 0$, hence $p\to \frac{1}{1+0}=1$. All mass on $a_1$ → deterministic/greedy. This is ordinary reward-only RL, matching the parent's "$\alpha \to 0$ ⇒ greedy." Degenerate tie $q_1=q_2$: then $\Delta =0$, so $\frac{\Delta }{\alpha }=0$ for every $\alpha >0$, giving $p=\frac{1}{1+e^0}=\frac{1}{2}$. The policy stays uniform $(0.5,0.5)$ regardless of temperature — entropy correctly refuses to break a genuine tie. Even as $\alpha \to 0$ this stays $0.5$ (the limit and the tie do not commute: you must fix $\Delta =0$ first).

Using $p=\frac{e^{3/\alpha }}{e^{3/\alpha }+e^{1/\alpha }}$ and $\mathcal{H}=-p\ln p-(1-p)\ln (1-p)$:

As $\alpha$ grows, $p\to 0.5$ and $\mathcal{H}\to \ln 2=0.69315$ (the maximum for two actions). Monotone increase confirmed. See the curve:

The student's claim, restated: once the Q-values look stable, set $\alpha =0$ (kill the entropy bonus everywhere) and treat SAC as "TD3 with a stochastic head" — expecting the same policy but with faster convergence from off-policy replay.

Reason 1 — the target is defined with entropy. The critic regresses toward $y=r+\gamma ({\min }_j{Q}_{{\bar{\theta }}_j}(s^{\prime },a^{\prime })-\alpha \log \pi (a^{\prime }|s^{\prime }))$. Drop the $-\alpha \log \pi$ term and you change the fixed point of the Bellman operator; the learned $Q$ no longer equals soft-Q, so the actor (which minimizes KL to $\exp (Q/\alpha )$) is now chasing an inconsistent target. This is not "TD3 but faster," it is a mismatched actor–critic pair whose two halves optimize different objectives.

Reason 2 — exploration collapse. As shown in Problem 5.1, without the entropy bonus the optimal policy is deterministic ($p\to 1$). That reintroduces exactly the brittleness (TD3/DDPG failure mode) SAC was designed to avoid: one bad Q estimate can lock the policy onto a wrong action with no self-correcting exploration. Auto-$\alpha$ (Problem 4.2) exists precisely so you don't have to hand-anneal entropy to zero.

Verdict: the claim is false — entropy is load-bearing for both the critic's target and the actor's continued exploration during training.

• Objective ::: $J(\pi )={\sum }_t\mathbb{E}[r+\alpha \mathcal{H}(\pi (\cdot |s_t))]$
• Soft value ::: $V(s)={\mathbb{E}}_a[Q(s,a)-\alpha \log \pi (a|s)]$
• Soft target ::: $y=r+\gamma ({\min }_j{Q}_{{\bar{\theta }}_j}(s^{\prime },a^{\prime })-\alpha \log \pi (a^{\prime }|s^{\prime }))$
• Optimal policy ::: ${\pi }^{\ast }\propto \exp (Q/\alpha )$ (Boltzmann)
• Temperature gradient ::: ${\nabla }_{\alpha }J={\mathcal{H}}_{\text{current}}-\bar{\mathcal{H}}$
• tanh correction ::: $\log \pi (a)=\log \mu (u)-\log (1-{\tanh }^{2}u)$ (raises $\log \pi$ since the term is $\le 0$)