Visual walkthrough — Soft Actor-Critic (SAC)

Before line one, three words must mean something. Let me build them with pictures, not jargon.

---

Step 0 — The three words we need (policy, entropy, Q)

WHAT. Three objects live in this story:

• A policy $\pi (a\mid s)$ — read "pi of a given s". It is a recipe for choosing: given a situation $s$ (a "state"), it hands you a probability for each possible action $a$. A probability is just a number between $0$ and $1$; all the actions' numbers must add up to $1$ (you will do something).
• The entropy $\mathcal{H}$ of that recipe — a single number measuring how spread out the choice is. All-eggs-in-one-basket → entropy $0$. Perfectly torn between options → entropy is large.
• The Q-value $Q(s,a)$ — a score saying "how good is action $a$ in state $s$, counting all future reward". Bigger $Q$ = better long-term move.

WHY these three. SAC's whole objective is "pick good actions ($Q$ high) and stay unpredictable (entropy $\mathcal{H}$ high)". You cannot talk about the trade-off until both are pictures.

PICTURE. Below: three bar charts. Left, a policy that dumps all probability on one action (sharp, low entropy). Right, a flat policy (spread, high entropy). The height of each bar is a probability; the bars sum to $1$.

---

Step 1 — Write down what we want to maximize

WHAT. At a single state $s$, we want the policy that scores highest on $J(\pi )=\underset{\text{average goodness}}{\underbrace{{\mathbb{E}}_{a\sim \pi }[Q(s,a)]}}+\underset{\text{surprise bonus}}{\underbrace{\alpha \mathcal{H}(\pi )}}.$

Term by term:

• ${\mathbb{E}}_{a\sim \pi }[\cdots ]$ means "average, weighting each action by how often the policy picks it". So ${\mathbb{E}}_{a\sim \pi }[Q]={\sum }_a\pi (a)Q(s,a)$.
• $\alpha$ (Greek "alpha") is the temperature: a positive dial for how much we care about surprise vs. goodness.
• $\mathcal{H}(\pi )$ is the entropy from Step 0.

WHY this exact sum. This is the Maximum Entropy RL idea in miniature: reward-only RL uses just the first term and collapses to one action; adding $\alpha \mathcal{H}$ pays the agent to hedge. We are solving the one-state version of the parent's objective $J(\pi )={\sum }_t\mathbb{E}[r+\alpha \mathcal{H}]$.

PICTURE. Two competing forces on the same policy: the $Q$-term is a magnet pulling all probability onto the best action; the entropy term is a spring pushing bars back toward flat. The winner of this tug-of-war is the optimal policy.

---

Step 2 — Add the "must be a probability" constraint

WHAT. We cannot just maximize $J$ freely: the bars are probabilities, so they must obey ${\sum }_a\pi (a)=1,\pi (a)\ge 0.$

WHY. Without this rule the entropy magnet would just make every bar enormous (more surprise forever). The constraint that the heights add to $1$ is what forces a genuine shape, not just "bigger everything".

PICTURE. Think of the probabilities as water poured across buckets whose total is fixed at $1$ litre. You can slosh water between buckets, but never create or destroy it. The optimization = the best way to distribute a fixed litre.

---

Step 3 — The trick for "maximize with a constraint": Lagrange

WHAT. To maximize $J$ while keeping $\sum \pi =1$, we build one combined quantity, the Lagrangian: $L={\sum }_a\pi (a)(Q(s,a)-\alpha \log \pi (a))+\lambda ({\sum }_a\pi (a)-1).$

Term by term:

• The first sum is exactly $J$ rewritten (since $\alpha \mathcal{H}=-\alpha \sum \pi \log \pi$).
• $\lambda$ (Greek "lambda") is a new helper number, one price tag attached to the "must sum to $1$" rule.
• The bracket $(\sum \pi -1)$ is zero when the rule holds — so at the valid answer, $L$ equals $J$.

WHY this tool and not plain calculus? If we ignored the constraint and just set the derivative of $J$ to zero, the answer would drift off the "sums to $1$" surface. Lagrange's method is the standard machine for "optimize on a curved constraint": it lets us do free calculus on $L$, and the $\lambda$ term automatically drags the answer back onto the constraint. It answers the question "where is the peak, given we're trapped on the sums-to-one sheet?"

PICTURE. The constraint is a slanted plane (all points that sum to $1$). $J$ is a hill. The best allowed point is where the hill's slope points straight out of the plane — that "straight-out" direction is precisely what $\lambda$ measures.

---

Step 4 — Take the derivative and set it to zero

WHAT. We nudge one bar $\pi (a)$ and ask "did $L$ change?" At the top, it doesn't — the derivative is zero: $\frac{\partial L}{\partial \pi (a)}=Q(s,a)-\alpha \log \pi (a)-\alpha +\lambda =0.$

Where each piece came from:

• $Q(s,a)$ — derivative of the $\pi Q$ term.
• $-\alpha \log \pi (a)$ — derivative of $-\alpha \pi \log \pi$ gives $-\alpha \log \pi -\alpha$ (product rule on $\pi \log \pi$; the extra $-\alpha$ is the "$-\alpha \cdot 1$" from $\frac{d}{d\pi }(\pi \log \pi )=\log \pi +1$).
• $+\lambda$ — derivative of the constraint term.

WHY "set derivative to zero"? The derivative is the slope — how fast $L$ climbs as we move probability onto action $a$. If any bar had positive slope we could pour more water there and score higher; if negative, pour less. Equilibrium (the peak) is exactly where every bar has the same, zero, marginal slope. That is the mathematical fingerprint of an optimum.

PICTURE. A ball resting at the bottom of a valley: at rest, tilt the floor any direction and the ball doesn't roll — slope zero. We freeze the policy at that no-roll point.

---

Step 5 — Solve for the policy: an exponential of Q appears

WHAT. Rearrange Step 4 for $\log \pi (a)$: $\alpha \log \pi (a)=Q(s,a)+(\lambda -\alpha )⟹\log \pi (a)=\frac{Q(s,a)}{\alpha }+\text{const}.$ Exponentiate both sides (undo the log) and let normalization absorb the constant: $\boxed{{\pi }^{\ast }(a\mid s)=\frac{\exp (Q(s,a)/\alpha )}{Z(s)}},Z(s)={\sum }_{a^{\prime }}\exp (Q(s,a^{\prime })/\alpha ).$

Term by term:

• $\exp (Q/\alpha )$ — the taller the $Q$, the exponentially taller the bar. Dividing by $\alpha$ stretches or squashes how sharply $Q$ differences matter.
• $Z(s)$ — the normalizer ("partition function"): the sum of all the un-normalized heights, dividing so the bars total $1$ (this is where our constraint from Step 2 gets satisfied).

WHY the exponential and nothing else? Because we set $\log \pi$ proportional to $Q$ in Step 5, and the only function whose log is linear in $Q$ is the exponential. The entropy term is what forced a $\log$; solving the $\log$ hands us $\exp$. This exact object is the Boltzmann / softmax policy, and it's identical to the one in Soft Q-Learning.

PICTURE. Feed a bar chart of $Q$-values through the machine $x\mapsto e^{x/\alpha }$, then normalize. Watch the tallest $Q$ balloon and the rest shrink — but never fully vanish (that residual is the surviving exploration).

---

Step 6 — Edge case: $\alpha \to 0$ (the greedy limit)

WHAT. Shrink the temperature toward zero. Then $Q/\alpha$ blows up, and $\exp (Q/\alpha )$ for the single best action dwarfs everything else. The policy becomes a spike on $\arg {\max }_{a}Q$ — a deterministic, greedy policy.

WHY it matters. This recovers ordinary reward-only RL — exactly the behaviour of DDPG / TD3. It confirms our formula is consistent: turn surprise off, get the classic greedy answer. It's also why over-small $\alpha$ kills exploration.

PICTURE. Same $Q$ bars, but $\alpha =0.1$: one bar reaches the ceiling, all others flatten to the floor.

---

Step 7 — Edge case: $\alpha \to \infty$ (the uniform limit)

WHAT. Grow the temperature large. Then $Q/\alpha \to 0$ for every action, so $\exp (Q/\alpha )\to 1$ for all — every bar equal. The policy becomes uniform: pure, indiscriminate exploration.

WHY it matters. It shows the other end of the dial. Between the Step 6 spike and this flat line lies every useful SAC policy. This continuum is precisely what automatic temperature tuning walks along, nudging $\alpha$ to hit a target entropy $\bar{\mathcal{H}}$.

PICTURE. Same $Q$ bars, $\alpha =50$: all bars melt to the same height regardless of $Q$.

---

The one-picture summary

WHAT. One figure compresses the whole derivation: constraint → Lagrange peak → $\log \pi \propto Q$ → ${\pi }^{\ast }\propto e^{Q/\alpha }$, with the two limits $\alpha \to 0$ (greedy) and $\alpha \to \infty$ (uniform) shown as the two ends of one slider.

Recall Feynman: the whole walkthrough in plain words

We wanted a way to pick actions that are good but not predictable. "Good" is the $Q$-score; "unpredictable" is entropy, the spread of our choice. We wrote one scoreboard adding the two, then remembered our choice-probabilities must add to $1$ litre of water. To find the best split of that litre we used Lagrange's trick — attach a price $\lambda$ to the "adds to one" rule so we can do plain calculus. Setting the slope to zero everywhere (the ball-at-rest condition) told us $\log$ of each probability grows straight-line with its $Q$-score. Undoing the log turns "straight line" into "exponential", so the best policy is $\exp (Q/\alpha )$ divided by a normalizer. Turn the temperature $\alpha$ tiny and it becomes a spike on the single best move (greedy); turn it huge and every move is equally likely (pure exploring). The temperature is literally the slider between certainty and curiosity.

Related tools used here: Bellman Equation (where these $Q$-values ultimately come from), the Reparameterization Trick (how real SAC differentiates through samples of this policy), and the stability contrast with PPO.