Soft Actor-Critic (SAC): Worked examples — AI-ML deep dive

This is a do-the-numbers companion to Soft Actor-Critic (SAC). The parent note tells you what the soft Bellman equations are; here we grind through every kind of input those equations can meet — every sign of the log-probability, the extreme temperatures $α \to 0$ and $α \to \infty$ , the degenerate deterministic policy, the tanh-squashing trap, and an exam-style twist. If you can reproduce all of these by hand, you understand SAC operationally.

Before we compute anything, five symbols must be earned (the parent used them; we re-anchor them so line one is readable):

Definition The five quantities we keep reusing

$r$ — the reward: a plain number the environment hands the agent for one $(s, a)$ transition. It can be positive (good — "you touched the cup"), negative (bad — "you knocked it over"), or zero (nothing happened). There is no fixed range; the designer chooses the scale. It is the un-discounted payoff for the current step only.
$γ$ — the discount factor, a number in $[0, 1)$ that shrinks future value: a reward $k$ steps ahead is worth $γ^{k}$ times as much now. We use $γ$ close to $1$ (e.g. $0.9$ – $0.99$ ) so the agent is far-sighted; $γ = 0$ would make it care only about the immediate reward. It multiplies the whole future block $V (s^{'})$ in every Bellman backup.
$Q (s, a)$ — a number: "how good is taking action $a$ in state $s$ , counting all future soft rewards." Bigger = better.
$lo g π (a ∣ s)$ — the natural logarithm of the probability the policy assigns to the action it took. Probabilities live in $(0, 1]$ , and $lo g$ of a number below $1$ is negative. So $lo g π$ is almost always $\leq 0$ ; it is $0$ only when the policy is certain ( $π = 1$ ).
$α > 0$ — the temperature, the exploration knob (see Maximum Entropy RL). The recurring building block is the entropy bonus $- α lo g π$ . Since $lo g π \leq 0$ , the term $- α lo g π \geq 0$ : a bonus you add that grows the more uncertain (spread-out) the policy is.

The scenario matrix

Every soft-Bellman / policy computation you'll ever be asked lands in one of these cells. The examples that follow are labelled with the cell they cover. The figure just below draws the single relationship that powers half of them: how the bonus $- α lo g π$ climbs as the policy grows unsure.

Cell	Case class	What's extreme / tricky	Example
A	Ordinary soft target $y$	positive reward, moderate entropy	Ex 1
B	Confident policy	$lo g π \to 0$ , bonus vanishes	Ex 2 part B
C	Very uncertain policy	$lo g π ≪ 0$ , bonus dominates	Ex 2 part C
D	Twin-Q disagreement	which Q wins the $min$ , both orderings	Ex 3
E	Temperature limit $α \to 0$	Boltzmann → greedy (deterministic)	Ex 4
F	Temperature limit $α \to \infty$	Boltzmann → uniform	Ex 4
G	tanh squashing correction	density change of variables	Ex 5
H	Automatic $α$ update sign	entropy below/above target	Ex 6
I	Real-world word problem	translate a task into $y$	Ex 7
J	Exam twist	negative reward + terminal state ( $γ$ -cutoff)	Ex 8

The red curve below is the entropy bonus $- α lo g π$ as a function of $lo g π$ (at $α = 0.5$ ). Read it left-to-right: at the far right ( $lo g π = 0$ , a certain policy) the bonus is zero; sliding left toward $lo g π = - 4$ (a very spread policy) the red curve rises. Every "Case B vs Case C" contrast in this page is just two points on this one line.

Example 1 — the ordinary soft target (Cell A)

Forecast: Guess whether $y$ is bigger or smaller than $3.0$ , and by roughly how much.

Take the pessimistic Q: $min_{j} Q = min (3.0, 3.4) = 3.0$ (recall $min_{j}$ = smaller of the two twin critics). Why this step? Like TD3, the smaller of the twin Qs fights overestimation bias (the $max$ in a single Q over-reports).
Entropy bonus: $- α lo g π = - 0.5 \times (- 0.7) = + 0.35$ . Why this step? This is the $- α lo g π$ piece of the soft value $V (s^{'})$ (parent Step 1). It is positive because $lo g π < 0$ .
Soft next-state value: $V (s^{'}) = 3.0 + 0.35 = 3.35$ . Why this step? $V (s^{'}) = E_{a^{'}} [Q - α lo g π]$ ; here we use the sampled $a^{'}$ .
Discount and add reward: $y = r + γ V (s^{'}) = 0.5 + 0.9 \times 3.35 = 0.5 + 3.015 = 3.515$ . Why this step? Soft Bellman Equation backup: immediate reward plus discounted soft future.

Verify: $y = 3.515 > 3.0$ , exactly as forecast — the entropy bonus $0.35$ pushed the future value above the raw $min$ -Q, then discounting by $0.9$ shrank it slightly, and $+ 0.5$ reward pulled it back up. Sanity: dropping the bonus gives $0.5 + 0.9 \times 3.0 = 3.2$ , and $3.515 - 3.2 = 0.315 = 0.9 \times 0.35$ ✓.

Example 2 — how confident vs uncertain policies change the bonus (Cells B & C)

Forecast: In which case is $y$ larger? Why should "less sure" mean "more valuable"?

Part B (Cell B) bonus: $- 0.5 \times (- 0.01) = + 0.005$ . Why this step? $π \approx 1$ means near-zero entropy, so almost no exploration reward.
Part B (Cell B) target: $y_{B} = 0 + 1 \times (2.0 + 0.005) = 2.005$ .
Part C (Cell C) bonus: $- 0.5 \times (- 4.0) = + 2.0$ . Why this step? A very spread policy has high entropy; SAC pays a big bonus for it.
Part C (Cell C) target: $y_{C} = 0 + 1 \times (2.0 + 2.0) = 4.0$ .

Verify: $y_{C} > y_{B}$ ✓ — the uncertain policy (Cell C) is worth twice as much here purely from the entropy term. This is the self-annealing exploration mechanism: high-entropy states get inflated value, so the critic keeps them attractive until Q evidence overwhelms the bonus. Both $y_{B}$ and $y_{C}$ are just two heights on the red curve of the scenario-matrix figure.

Example 3 — twin-Q disagreement, both orderings (Cell D)

Forecast: Does swapping which network is bigger change $y$ ?

Bonus (shared): $- 0.2 \times (- 1.0) = + 0.2$ .
Case (i): $min_{j} Q = min (10, 6) = 6 \Rightarrow V = 6 + 0.2 = 6.2 \Rightarrow y = 1 + 0.99 \times 6.2 = 1 + 6.138 = 7.138$ . Why this step? $min_{j}$ ignores the optimistic $10$ ; it always trusts the stricter judge.
Case (ii): $min_{j} Q = min (6, 10) = 6 \Rightarrow$ identical $V = 6.2 \Rightarrow y = 7.138$ .

Verify: Both orderings give the same $y = 7.138$ ✓. The $min_{j}$ is symmetric — only the value of the smaller Q matters, never its index $j$ . This is the point of clipped double-Q: the optimistic estimate ( $10$ ) is discarded no matter which head produced it.

Example 4 — the two temperature limits (Cells E & F)

Forecast: Sketch the bar chart of $π$ for each $α$ before computing. Which becomes a single spike? Which becomes flat?

$α = 1$ : weights $e^{2}, e^{1}, e^{0.5} = 7.389, 2.718, 1.649$ , sum $= 11.756$ . $π = (0.6286, 0.2312, 0.1402)$ . Why this step? Moderate temperature → clear preference but real probability mass on all actions.
$α = 0.05$ : exponents $40, 20, 10$ → the $e^{40}$ term dominates by factors of $e^{20}$ and $e^{30}$ . $π \approx (1, 2 \times 1 0^{- 9}, \approx 0)$ — essentially $π = (1, 0, 0)$ . Why this step? Small $α$ = greedy: this is exactly reward-only RL / DDPG behaviour (Cell E limit).
$α = 20$ : exponents $0.1, 0.05, 0.025$ → $e^{0.1}, e^{0.05}, e^{0.025} = 1.1052, 1.0513, 1.0253$ , sum $= 3.1818$ . $π = (0.3474, 0.3304, 0.3222)$ . Why this step? Large $α$ = uniform, maximal exploration (Cell F limit).

Verify: Each row sums to $1$ (the normalizer $Z$ guarantees this). $α = 0.05$ collapses to a one-hot vector; $α = 20$ is nearly flat $(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ ✓. This reproduces the parent's claim: $α \to 0$ ⇒ greedy, $α \to \infty$ ⇒ uniform. Compare with Soft Q-Learning, where this same Boltzmann form is the policy itself.

The figure below draws all three probability vectors as bar groups; the red bars are the greedy $α = 0.05$ case — notice the single red spike on action $a_{1}$ while the faded moderate/uniform cases spread mass across all three actions.

Example 5 — the tanh squashing correction (Cell G)

Forecast: Will the correction make $lo g π$ larger (closer to $0$ ) or smaller (more negative) than the raw Gaussian value?

Raw Gaussian log-density at $u = 0.5$ : $lo g N (u) = - \frac{1}{2} lo g (2 π) - \frac{u ^{2}}{2} = - 0.9189 - 0.125 = - 1.0439$ . Why this step? This is the density of the pre-squash variable, using the Reparameterization Trick sample.
Squash Jacobian term: the change of variables is $lo g π (a) = lo g N (u) - lo g (1 - tanh^{2} u)$ . With $tanh (0.5) = 0.46212$ , $tanh^{2} = 0.21356$ , so $1 - tanh^{2} = 0.78644$ and $- lo g (0.78644) = + 0.24011$ . Why this step? $a = tanh u$ compresses the real line into $(- 1, 1)$ ; a change of variables multiplies the density by $\frac{d u}{d a} = \frac{1}{1 - t a n h ^{2} u}$ . In log-space that adds $- lo g (1 - tanh^{2} u)$ , a positive number.
Corrected log-prob: $lo g π (a) = lo g N (u) + (- lo g (1 - tanh^{2} u)) = - 1.0439 + 0.2401 = - 0.8038$ . Why this step? Forgetting the $- lo g (1 - tanh^{2} u)$ term (the parent's [!mistake]) would report $- 1.0439$ and corrupt entropy and $α$ -tuning.

Verify: The correction added $+ 0.2401$ , so the corrected $lo g π = - 0.8038$ is larger (closer to $0$ ) than the raw $- 1.0439$ ✓. That is the expected direction: since $- lo g (1 - tanh^{2} u) \geq 0$ always, the correction can only raise $lo g π$ toward $0$ — never below the raw Gaussian value. (Intuition check: near the flat edges of $tanh$ , a wide band of $u$ maps to a narrow band of $a$ , concentrating probability, which raises the density there.)

Example 6 — the automatic- $α$ update sign (Cell H)

Forecast: Current entropy is $H = - E [lo g π] = + 1.0$ ; the target entropy magnitude is $∣ \overset{ˉ}{H} ∣ = 3$ . We are below target — intuitively, should $α$ grow (explore more) or shrink?

Write the gradient of the loss w.r.t. $α$ . $α$ appears linearly in $J (α) = - α (lo g π + \overset{ˉ}{H})$ , so $\frac{dJ}{d α} = - (lo g π + \overset{ˉ}{H}) = - (- H + \overset{ˉ}{H}) = H - ∣ \overset{ˉ}{H} ∣,$ using $lo g π = - H$ and $\overset{ˉ}{H} = - ∣ \overset{ˉ}{H} ∣$ . Why this step? This single, clean expression $\frac{dJ}{d α} = H - ∣ \overset{ˉ}{H} ∣$ is the entire sign story: it compares current entropy to the target magnitude, nothing else.
Apply gradient descent: $α \leftarrow α - η \frac{dJ}{d α} = α - η (H - ∣ \overset{ˉ}{H} ∣)$ . Why this step? We minimize $J$ , so we step against the gradient. Read the sign of $(H - ∣ \overset{ˉ}{H} ∣)$ to get the direction of $α$ .
Under-explored case ( $lo g π = - 1.0 \Rightarrow H = 1.0$ ): $\frac{dJ}{d α} = 1.0 - 3 = - 2.0 < 0$ , so $α \leftarrow α - η (- 2.0) = α + 2 η$ — $α$ increases. Why this step? Entropy is below target, so raise the exploration knob. Matches intuition ✓.
Over-explored case ( $lo g π = - 5.0 \Rightarrow H = 5.0$ ): $\frac{dJ}{d α} = 5.0 - 3 = + 2.0 > 0$ , so $α \leftarrow α - η (+ 2.0) = α - 2 η$ — $α$ decreases. Why this step? Too much entropy, so shrink the knob. Also matches intuition ✓.

Verify: The fixed point is $\frac{dJ}{d α} = 0 \Rightarrow H = ∣ \overset{ˉ}{H} ∣ = 3$ : $α$ stops moving exactly when measured entropy equals the target. Under-explored ( $H = 1 < 3$ ) raises $α$ ; over-explored ( $H = 5 > 3$ ) lowers it ✓. So gradient descent on $J (α)$ automatically self-anneals exploration — the sign works out to the intuitive direction in both cases.

Recall Clean rule to memorize

Sign rule ::: $\frac{dJ}{d α} = H - ∣ \overset{ˉ}{H} ∣$ . Gradient descent then raises $α$ when entropy $<$ target, lowers it when entropy $>$ target. Fixed point: entropy $= ∣ \overset{ˉ}{H} ∣$ .

Example 7 — real-world word problem (Cell I)

Forecast: The many-good-angles fact should raise $y$ through the bonus. Estimate before computing.

Pessimistic Q: $min_{j} Q = min (8.0, 7.5) = 7.5$ . Why this step? Overestimation control — trust the stricter critic.
Entropy bonus: $- 0.3 \times (- 2.5) = + 0.75$ . Why this step? "Many equally good arm angles" = high entropy = large bonus, exactly the situation SAC rewards.
Soft next value: $V = 7.5 + 0.75 = 8.25$ .
Backup: $y = 2 + 0.95 \times 8.25 = 2 + 7.8375 = 9.8375$ . Why this step? Reward for touching now, plus discounted flexible future.

Verify: Without the flexibility bonus, $y$ would be $2 + 0.95 \times 7.5 = 9.125$ . The extra $0.7125 = 0.95 \times 0.75$ credits the robot for having many working strategies ✓ — a concrete instance of the parent's candy-robot intuition.

Example 8 — exam twist: negative reward at a terminal state (Cell J)

Forecast: At a terminal state, what happens to the whole $γ (\dots)$ block? Does the entropy bonus survive?

Terminal mask: the bootstrap term is multiplied by $(1 - d)$ with $d = 1$ (done). So $y = r + γ (1 - d) [\dots] = r + 0 = - 1$ . Why this step? There is no future after termination, so both the future Q and the entropy bonus vanish — you cannot be paid for future exploration that never happens.
Non-terminal contrast ( $d = 0$ ): bonus $= - 0.4 \times (- 0.5) = + 0.2$ ; $V = 3.0 + 0.2 = 3.2$ ; $y = - 1 + 0.99 \times 3.2 = - 1 + 3.168 = 2.168$ . Why this step? Same negative reward, but now the future exists, so the bonus and discounted Q reappear.

Verify: Terminal $y = - 1$ (pure reward, no bonus) vs non-terminal $y = 2.168$ ✓. The exam trap is forgetting the $(1 - d)$ mask and leaving an entropy bonus on a terminal transition — that would leak fictitious future value into a state that has no future. Note the entropy bonus is inside the $γ$ block, so it is correctly zeroed by termination.

See also PPO for the on-policy contrast the parent draws, and Reparameterization Trick for how Ex 5's sample lets gradients flow.