Question bank — Soft Actor-Critic (SAC)

False. As $\alpha \to 0$ the Boltzmann policy ${\pi }^{\ast }\propto \exp (Q/\alpha )$ collapses onto the single highest-$Q$ action — it becomes deterministic, i.e. plain reward-only RL like DDPG.

False. Entropy is a means, not the goal. Too-large $\alpha$ makes the policy near-uniform and it never commits to the good actions, so task reward drops. The bonus helps exploration, not the score itself.

True. $V={\mathbb{E}}_a[Q]-\alpha {\mathbb{E}}_a[\log \pi ]$, and the $-\alpha \log \pi$ entropy term is positive (since $\log \pi <0$), so $V$ can sit above the best Q by up to the entropy bonus.

False. Expressiveness is irrelevant; the twin is about bias. A single Q self-selects noisy over-estimates through the target's max-like step, so we take $\min (Q_1,Q_2)$ to stay pessimistic — the same fix as TD3.

True. Off-policy learning updates the current policy from data generated by earlier policies, so nothing is thrown away; on-policy PPO must discard data after each update.

True. In the target it becomes the $V(s^{\prime })$ bonus ($-\alpha \log \pi (a^{\prime }|s^{\prime })$); in the actor loss the $\alpha \log \pi (a|s)$ term is what rewards the policy for staying random. Same idea, two places.

False. It is tuned toward a target entropy $\bar{\mathcal{H}}$ (often $-\dim (\mathcal{A})$). If current entropy is below target, $\alpha$ grows to encourage more exploration; if above, it shrinks.

False. SAC uses the Reparameterization Trick: $a=\tanh ({\mu }_{\varphi }+{\sigma }_{\varphi }\odot ϵ)$ makes the action a differentiable function of $\varphi$, so gradients flow straight through the sample — much lower variance than the score-function estimator.

We squash with $a=\tanh (u)$, and a change of variables changes the density. The correct log-prob subtracts the Jacobian term: $\log \pi (a)=\log \mathcal{N}(u)-{\sum }_i\log (1-{\tanh }^2{u}_i)$.

The opposite — SAC uses ${\min }_j$. Optimism amplifies overestimation bias; the pessimistic minimum cancels it, which is the whole point of clipped double-Q.

The sign is negative: $-\alpha \log \pi (a^{\prime }|s^{\prime })$. Because $\log \pi <0$, the minus makes the term positive, correctly adding value for a more random next-state policy.

The correction $-\sum \log (1-{\tanh }^2{u}_i)$ depends on $u$, not a constant. Omitting it biases the entropy estimate action-by-action, corrupting both the actor gradient and the auto-$\alpha$ update.

The actor minimizes $\alpha \log \pi -Q$. Minimizing $\log \pi$ means making it more negative, i.e. higher entropy. Maximizing $\log \pi$ would make the policy spiky — the reverse of what we want.

Partly right in spirit, but the augmented ("soft") reward $r+\alpha \mathcal{H}$ re-derives soft Bellman equations where $V$ carries the entropy of all future steps — see Bellman Equation and Maximum Entropy RL.

It is $\exp (Q/\alpha )$, i.e. divide by $\alpha$. This matters: small $\alpha$ sharpens the exponent (greedy), large $\alpha$ flattens it (uniform) — the opposite of what $\exp (\alpha Q)$ would give.

Maximizing pure reward puts all probability on the single best action, so one bad Q estimate reroutes the policy forever — no hedge, no exploration. Entropy keeps near-equal actions alive as a cushion.

With auto-$\alpha$, once the policy has learned confident good actions its entropy naturally drops toward the target and $\alpha$ shrinks, so exploration fades exactly when it is no longer needed — no manual schedule.

Because exploration must be encouraged throughout the trajectory. The soft $V$ recursion accumulates the entropy bonus of all future steps, so acting randomly-but-usefully now is valued the whole way down.

Clipping is non-differentiable and piles probability mass on the boundaries, breaking the log-prob. $\tanh$ is smooth and invertible, so the reparameterization gradient and the change-of-variables density both stay well-defined.

DDPG's deterministic actor overfits sharp Q peaks and chases exploitable errors; SAC's entropy term smooths the target landscape and its twin critics curb overestimation, damping the feedback loops that make DDPG diverge.

Soft Q-Learning shows the entropy-regularized greedy policy is the Boltzmann distribution $\exp (Q/\alpha )/Z$; SAC's actor is simply a tractable neural approximation that minimizes KL to that same target.

The exponent $Q/\alpha \to 0$ for every action, so $\exp (Q/\alpha )\to 1$ uniformly — the policy becomes uniform random, pure exploration with no reward preference.

$\mathcal{H}=0$ (no uncertainty). SAC can only approach it as $\alpha \to 0$; with $\alpha >0$ and a Gaussian actor the entropy stays strictly positive, so it never fully commits.

No — $\min (Q,Q)=Q$, so the clip is inactive. The twin only helps when the networks disagree, which is exactly when one is likely an over-optimistic outlier.

There $1-{\tanh }^{2}u\to 0$, so $-\log (1-{\tanh }^{2}u)\to +\infty$ — the density blows up and log-probs become numerically unstable. Implementations add a small $\epsilon$ inside the log to stay finite.

Yes — the soft Bellman target only needs the transition $(s,a,r,s^{\prime })$ plus a fresh action $a^{\prime }\sim {\pi }_{\varphi }$ at $s^{\prime }$. Being off-policy, it does not care which policy generated the stored $(s,a)$.

No — you cannot reparameterize a categorical sample through $\tanh$. Discrete SAC instead sums the actor/entropy terms over all actions analytically, sidestepping the sampling gradient entirely.