Intuition The one core ideaA normal agent chases the single best action and freezes there; SAC pays the agent a bonus for staying a little bit random , so it keeps exploring and never gets stuck betting everything on one guess. Everything else — the "soft" values, the two critics, the temperature knob — is just the bookkeeping needed to reward reward-plus-randomness at every step.
This page assumes you have seen nothing . We will name every letter and squiggle the Soft Actor-Critic (SAC) note throws at you, draw the picture behind it, and say why the topic can't live without it. Read top to bottom — each item is a brick for the next.
Before any math, picture the game being played.
Definition The four everyday words
State s sees about the world (robot's joint angles, ball position). Picture: a snapshot photo of the situation.Action a chooses right now (a torque, a steering angle). Picture: the arrow the agent pushes on the world.Reward r ( s , a ) Next state s ′ after the action lands. The prime mark ′
Intuition Why we need theseEvery RL method — from DDPG to PPO — is a recipe for turning states into good actions. SAC is no exception; it just adds a twist to how good is measured. You cannot read one line of the topic without s , a , r , s ′
The action lives in a set called the action space A dim ( A ) dim ( A ) = 6
A deterministic agent has a rule "in state s a stochastic policy : a whole probability cloud over actions.
π ( a ∣ s ) π ( a ∣ s ) probability (really, probability density ) that the agent picks action a s ∣ "given" : "chance of a given we are in s
Picture: a bell-shaped hill sitting over the action axis. Tall where the action is likely, flat where it is unlikely.
It always sums (integrates) to 1: ∫ π ( a ∣ s ) d a = 1
Intuition Why a cloud and not an arrow?A single arrow, once wrong, is wrong forever. A cloud lets the agent sample — sometimes the best action, sometimes a neighbour. That built-in wiggling is exploration, and it is the heart of SAC. Compare this with TD3 /DDPG , which use an arrow plus hand-added noise; SAC bakes the randomness into π
The subscript π ϕ ϕ ϕ
The letters log π ( a ∣ s )
log x log x "e x Here e ≈ 2.718 log ( e ) = 1 log ( 1 ) = 0 0.01 log ( 0.01 ) ≈ − 4.6
Intuition Why log, and why this tool specifically?Probabilities are tiny and multiply awkwardly. We want a "surprise meter": common events → small surprise, rare events → large surprise. The quantity − log π ( a ∣ s )
If π = 1 = − log 1 = 0
If π − log π
So − log π ( a ∣ s ) "how surprised am I that I chose this action?" That single quantity is the seed of entropy in the next section.
Now average that surprise over all the actions the policy might take.
Intuition The picture of entropy
A sharp spike (one action almost certain) → surprise nearly 0 → low entropy .
A flat spread (many actions equally likely) → lots of surprise → high entropy .
Entropy is a single knob-reading for "how undecided is the policy." SAC pays the agent for keeping this high. See Entropy (Information Theory) and Maximum Entropy RL for the deeper story.
The agent cares not just about this reward but the whole future stream. Two bookkeepers track that.
Definition The two value functions
State value V ( s ) s Action value Q ( s , a ) commit to action a (state, action) pair.
Definition Discount factorγ γ 0.99 shrinks future rewards : a reward n γ n
The soft versions used by SAC just glue the entropy bonus onto V V ( s ) = E a ∼ π [ Q ( s , a ) − α log π ( a ∣ s ) ] − α log π H soft .
α α positive number multiplying the entropy bonus. It sets how much the agent is paid for randomness.
α → 0 greedy/deterministic (ordinary reward-only RL).α → ∞ uniform (pure random exploration).In between: a graceful mix.
Intuition Why call it "temperature"?Borrowed from physics: hot systems jiggle everywhere (high α α π ∗ ∝ exp ( Q / α ) α Soft Q-Learning .
Real actuators saturate: a torque can't be infinite. SAC keeps actions inside [ − 1 , 1 ] tanh function.
tanh ( u ) tanh u ( − 1 , 1 ) u → + 1 u → − 1 tanh ( 0 ) = 0
ϵ ϵ ∼ N ( 0 , I ) a = tanh ( μ ϕ ( s ) + σ ϕ ( s ) ⊙ ϵ )
μ ϕ ( s ) σ ϕ ( s ) ⊙ Why do this? So the randomness lives in ϵ through the sample into ϕ Reparameterization Trick .
Common mistake The squash changes the densityBecause a = tanh ( u ) log π ( a ) not just the Gaussian log-density. You must subtract a correction ∑ i log ( 1 − tanh 2 ( u i )) α
entropy H is average surprise
soft values and soft Bellman
tanh squash and epsilon noise
Say each answer out loud before revealing — if any stalls, reread that section.
What does s ′ The next state, the snapshot right after the action lands.
What is π ( a ∣ s ) The probability (density) of choosing action a s
Read the bar ∣ "given" — the thing after it is the condition we assume.
What is − log π ( a ∣ s ) The "surprise" of that action — near 0 for likely actions, large for rare ones.
What is entropy H ( π ( ⋅ ∣ s )) The average surprise, E a ∼ π [ − log π ]
Spike-shaped policy vs flat policy: which has higher entropy? The flat one (many actions equally likely) has higher entropy.
What does E a ∼ π [ ⋅ ] The weighted average of the bracketed quantity over actions drawn from π
Difference between V ( s ) Q ( s , a ) V Q
Why is γ To shrink far-future rewards so totals stay finite; 0 ≤ γ < 1
What does the temperature α Reward vs entropy — small α α
Why squash actions with tanh To keep actions bounded in ( − 1 , 1 )
What does the reparameterization trick let us do? Move randomness into fixed noise ϵ ϕ
What correction does tanh force on log π Subtract ∑ i log ( 1 − tanh 2 ( u i ))
