Intuition The ONE core ideaTo teach a network good "move-scores," we ask it to match a target that is built from its own current guesses — like aiming at a bullseye that jerks every time you shoot. A target network is a slow-moving frozen copy of the network that holds the bullseye still long enough to actually hit it.
This page assumes you know nothing . Before you read the parent note we will earn every symbol it throws at you — s a r γ Q θ max ∇ τ
Reinforcement learning is about an agent (a decision-maker) living in an environment . At each moment it sees where it is, does something, and gets a numeric score. That is all.
s s snapshot of the situation the agent is in right now. Plain words: "everything the agent can see at this instant."
Picture: a single square on a board game, or a single frame of a video game.
Why the topic needs it: every value the network predicts is about a particular state . Without s
a a choice the agent can make from the current state (move left, jump, fire...).
Picture: an arrow leaving the current square toward a possible next square.
Why: the network scores actions , so a
r r single number the environment hands back right after an action: how good that immediate step was (points gained, + 1 − 1 Picture: a little green + 1 Why: rewards are the ground truth — the only honest signal telling us what "good" means.
s ′ s ′ state you land in after doing action a s Picture: the square the arrow points to .
Why: to judge an action we must peek at where it takes us — that peek is what causes all the trouble later.
Look at the figure: state s a r s ′ ( s , a , r , s ′ ) transition — the atom of everything below.
We do not just want the immediate reward — a move that scores + 1 total future score . Two symbols make that precise.
Definition The discount factorγ γ number between 0 and 1 that shrinks future rewards. A reward k γ k Picture: a row of coins fading to grey the further into the future they are — near coins bright, distant coins dim.
Why this tool and not another? We need a way to say "future matters, but less than now" AND to keep the sum finite. Multiplying by a fraction each step does both: γ = 0.9 0. 9 10 ≈ 0.35
Definition The action-valueQ ( s , a ) Q ( s , a ) "how much total (discounted) future reward do I expect if I take action a s The letter Q Q uality.
Picture: a number floating over each outgoing arrow — the arrow's "worth."
Why: if we know Q Q is the whole game.
Definition The greedy max:max a ′ Q ( s ′ , a ′ ) Q a ′ s ′ max "take the biggest of these numbers." Here a ′ in the next state .
Picture: in s ′ max Why this tool? After landing in s ′ play well — and playing well means taking the best action. "Best" = biggest Q max
Here is the elegant trick (the Bellman Equation , covered fully there): the value of now equals the reward now plus the discounted value of the best next step .
Intuition Why this is the seed of all troubleLook at y contains Q made out of the same scores we are trying to learn . Adjust the scores → the target moves. This "chasing your own tail" is precisely what the target network exists to cure. Every symbol below exists to describe and fix this loop.
For tiny games we could store Q ( s , a ) neural network to approximate Q
Definition The parametersθ θ big bag of tunable numbers (weights) inside the network. Writing Q ( s , a ; θ ) ( s , a ) using the current knob settings θ Picture: a control panel full of dials; turning the dials changes every output number.
Why the notation: we must be able to say which knob-setting produced a value — because the fix (target networks) uses two different settings at once .
Definition The frozen copyθ − θ − saved, older snapshot of the knobs, kept aside and updated only slowly. The minus is just a label meaning "the lagged one."
Picture: a photograph of the control panel taken a while ago, taped next to the live panel.
Why: if the target is computed from this frozen photo, it holds still while the live panel learns — the bullseye stops jerking.
The figure shows the two panels: the live online network θ target network θ − blue panel.
L ( θ ) L ( θ ) how wrong the network is , as one number. We use the squared error : ( y − Q ( s , a ; θ ) ) 2 y Q Picture: a vertical red bar between the target dot and the prediction dot; its length-squared is the loss.
Why squared? Squaring makes every error positive (so over- and under-shoots don't cancel), punishes big mistakes more, and gives a smooth bowl-shaped landscape that is easy to roll downhill on.
∇ θ ∇ θ L arrow pointing in the direction that increases L , one component per knob. We step the opposite way to shrink L Picture: standing on a hillside of loss; ∇ Why this tool? With millions of knobs we cannot try settings one-by-one. The gradient tells us, for every knob at once, which way to nudge it — the only scalable way to learn.
Definition The learning rateα α step size : how far we move down the gradient each update.
Picture: the length of one footstep down the loss hill.
Why: too big and we leap past the bottom; too small and we crawl. Its relative size versus how fast θ −
Common mistake The subtle point the whole topic hinges onWhen the target uses the same θ ∇ θ L oscillation and divergence . By computing the target from frozen θ − pretend the target is a constant : no gradient flows through θ −
Definition The Polyak factorτ τ 0 < τ ≪ 1 θ − ← τ θ + ( 1 − τ ) θ − ≪ Picture: each step, the blue frozen panel takes one tiny step (τ Why: it lets the target drift smoothly instead of jumping. τ → 0 τ → 1 τ 0.005 1/ τ = 200
Worked example Sanity-check the soft updateτ = 0.01 θ = 10 θ − = 4 θ new − = 0.01 ( 10 ) + 0.99 ( 4 ) = 0.1 + 3.96 = 4.06 0.06 100 × is the stability.
Greedy max over next actions
Gradient nabla and step alpha
Test yourself — reveal only after answering.
What does a transition ( s , a , r , s ′ ) From situation s a r s ′
Why multiply future rewards by γ To value the future less than now AND keep the total finite; a reward k γ k
What does Q ( s , a ) The total discounted future reward from taking a s
What does max a ′ Q ( s ′ , a ′ ) The best next-action value in s ′
What is θ θ − θ θ −
Why square the error in the loss L Makes errors positive, penalises big mistakes more, and gives a smooth bowl to descend.
What does the gradient ∇ θ L The direction (per knob) that increases loss fastest; we step opposite to shrink it.
Does gradient flow through θ − No — it is treated as a constant, so the target is a fixed label.
What does τ Tracking speed; small τ ∼ 1/ τ τ → 1 τ → 0
Why does using the same θ The target y Q ( ⋅ ; θ ) θ
Hinglish version of the parent Bellman Equation — supplies the self-referential target y Deep Q-Networks (DQN) — where these symbols all come together.Experience Replay — the sibling stabiliser.Double DQN — reuses θ θ − DDPG / TD3 / SAC — soft-update (τ Deadly Triad — the divergence this all fights.Two-Timescale Stochastic Approximation — the theory behind slow θ − 
