Q-learning algorithm: Worked examples — AI-ML deep dive

Before we start, let's re-anchor the one machine we use over and over. Everything below is just this equation, run with different numbers:

The picture below is that same equation drawn as a number line — keep it in your head for every example.

Reading the figure. The blue dot on the left is our old belief $Q (s, a)$ . The yellow dot on the right is the TD target $r + γ max_{a^{'}} Q (s^{'}, a^{'})$ . The gap between them is the TD error $δ$ (red span). The update slides the blue dot a fraction $α$ of the way toward yellow — landing at the green dot. Big $α$ = long slide; $α = 1$ = lands exactly on yellow; $δ = 0$ = dots coincide, no motion. Every worked example is just "where do the two dots sit, and where does green land."

Let me define each letter once so nothing is unearned:

$s$ = the state I'm in right now (e.g. a square on a grid). Think "where am I."
$a$ = the action I just chose (e.g. "move right"). Think "what I did."
$r$ = the reward the environment handed me for doing $a$ in $s$ . A single number, can be positive, negative, or zero.
$s^{'}$ = the state I landed in after acting. Read "s-prime."
$a^{'}$ = a possible next action, used only inside the $max$ . We look at all of them and keep the best.
$α$ (alpha), the learning rate: how big a step I take, between 0 and 1. Small = cautious, big = jumpy.
$γ$ (gamma), the discount factor: how much I care about the future, between 0 and 1. 0 = only now, 1 = future matters as much as now.
$max_{a^{'}} Q (s^{'}, a^{'})$ = the largest Q-value among all actions available in the next state. It answers "if I play perfectly from $s^{'}$ onward, how good is that?"
$δ$ (delta), the TD error: defined as $δ = target - Q (s, a)$ — the gap between fresh evidence and old belief. It measures "how surprised was I?" The whole update is just " $Q \leftarrow Q + α δ$ ": nudge by alpha times the surprise. When $δ = 0$ , evidence agrees with belief and nothing changes.
$a^{*}$ (a-star), the optimal action: the action with the highest Q-value in a state, $a^{*} = ar g max_{a^{'}} Q (s^{'}, a^{'})$ . "Arg-max" means "the action that achieves the max," as opposed to $max$ which gives the max value. It answers "which move is best?" while $max$ answers "how good is the best move?"
$ε$ (epsilon), the exploration rate in == $ε$ -greedy==: a probability between 0 and 1. With probability $ε$ the agent picks a random action (explore); with probability $1 - ε$ it picks the greedy $a^{*}$ (exploit). Typically start high (e.g. $ε = 0.3$ ) and decay toward 0 as learning matures — see Exploration-Exploitation Trade-off.

The scenario matrix

Every Q-update falls into one (or more) of these cells. The examples below are each labelled with the cell they cover.

Cell	Scenario class	What's special about it
A	Next state is terminal	$max_{a^{'}} Q (s^{'}, a^{'}) = 0$ by definition — the future term vanishes
B	Zero reward ( $r = 0$ )	Nudge comes purely from the discounted future
C	Negative reward ( $r < 0$ )	Target can drop below old value; Q can go negative
D	$γ = 0$ (myopic)	Future is ignored entirely; Q collapses to immediate reward
E	$γ = 1$ (undiscounted)	Full future value; only safe in episodic tasks
F	$α = 1$ (full replace)	Old belief thrown away; Q becomes the target exactly
G	Ties in the max	Two next-actions equal — max still well-defined, pick either
H	Off-policy update (suboptimal action taken)	The $max$ still points at optimal future
I	Stochastic reward (maximization bias)	Averaging over noisy targets; where Double Q-learning helps
J	Real-world word problem	Translating a messy story into $(s, a, r, s^{'})$
K	Exam twist — multi-step backward propagation	Watch value flow backward across several updates

Nine of the eleven cells are single-line arithmetic; two (I, K) show emergent behaviour. Let's fill them all.

Worked examples

Forecast: Guess — does $Q (s, a)$ go up a little, up a lot, or down? Write your guess.

Identify the future term. $s^{'}$ is terminal, so by definition $max_{a^{'}} Q (s^{'}, a^{'}) = 0$ — there is no future after the episode ends. Why this step? Terminal states are seeded with $Q = 0$ precisely so the recursion stops. If we forgot this we'd add phantom future value.
Build the TD target. $target = r + γ \cdot 0 = 10 + 0 = 10$ . Why this step? The target is "reward now plus discounted best future" — with no future, it's just the reward.
Compute the TD error. $δ = target - Q (s, a) = 10 - 0.5 = 9.5$ . Why this step? The error is how surprised we are. We badly under-valued this move.
Nudge. $Q (s, a) \leftarrow 0.5 + 0.1 \times 9.5 = 0.5 + 0.95 = 1.45$ . Why this step? We move 10% of the way toward the target, not all the way — one lucky sample shouldn't overwrite everything.

Verify: Old value 0.5, target 10, we should land strictly between them and one-tenth of the gap up: $0.5 + 0.1 (10 - 0.5) = 1.45$ . ✓ It went up a lot but not to 10 — matching cell A behaviour.

Forecast: With zero reward, is the update dead (no change) or does it still move? Guess.

TD target. $target = 0 + 0.9 \times 6.0 = 5.4$ . Why this step? Even with no immediate reward, the discounted promise of a good next state (value 6) pulls this Q up.
TD error. $δ = 5.4 - 2.0 = 3.4$ .
Nudge. $Q (s, a) \leftarrow 2.0 + 0.1 \times 3.4 = 2.34$ . Why this step? Value flows backward from good states even across zero-reward steps — this is how paths get "wired" toward distant goals.

Verify: $2.0 + 0.1 (5.4 - 2.0) = 2.34$ . ✓ The update is alive — cell B proves reward $= 0$ does not freeze learning.

Forecast: Will $Q (s, a)$ stay at 0, go slightly negative, or crash way down?

TD target. $target = - 5 + 0.9 \times 0 = - 5$ . Why this step? Punishment now, nothing good ahead — the target is squarely negative.
TD error. $δ = - 5 - 0 = - 5$ .
Nudge with the larger $α = 0.5$ . $Q (s, a) \leftarrow 0 + 0.5 \times (- 5) = - 2.5$ . Why this step? A bigger learning rate means we believe this bad news faster. The agent will now avoid this square.

Verify: $0 + 0.5 (- 5) = - 2.5$ , negative as expected. ✓ Cell C confirmed: Q-values are signed, and half a step of $- 5$ lands us at $- 2.5$ .

Forecast: Which $γ$ pushes $Q$ higher, and by how much roughly?

$γ = 0$ target. $target = 3 + 0 \times 8 = 3$ . Update: $1 + 0.1 (3 - 1) = 1.2$ . Why this step? $γ = 0$ literally deletes the future term — the agent is a pleasure-seeking goldfish, caring only about the reward in front of it.
$γ = 1$ target. $target = 3 + 1 \times 8 = 11$ . Update: $1 + 0.1 (11 - 1) = 2.0$ . Why this step? $γ = 1$ counts the full future value (8) with no shrinkage. This is only safe in episodic tasks that end; in an endless task it can blow up to infinity.

Verify: myopic $= 1.2$ , undiscounted $= 2.0$ . The undiscounted update is bigger because it sees the reward ahead. ✓ Cells D and E covered — $γ$ is the dial between short-sighted and far-sighted.

Forecast: Does the tie cause trouble? Where does $Q (s, a)$ end up with $α = 1$ ?

Resolve the max despite the tie. $max_{a^{'}} Q (s^{'}, a^{'}) = max (6, 6, 3, 1) = 6$ . Why this step? $max$ returns a value, not an action — two actions sharing the top value 6 is fine, the value is unambiguously 6. Ties only matter when we later choose an action (pick either winner), not here.
TD target. $target = 2 + 0.5 \times 6 = 5$ .
Full replace with $α = 1$ . $Q (s, a) \leftarrow 4 + 1 \times (5 - 4) = 5$ . Why this step? With $α = 1$ the old value is completely discarded — $Q (s, a)$ becomes the target exactly. Great for deterministic worlds, terrible for noisy ones (one bad sample overwrites everything).

Verify: $4 + 1 (5 - 4) = 5$ , which equals the target exactly. ✓ Cell F: $α = 1 \Rightarrow Q = target$ . Cell G: the tie is harmless — $max$ still returns 6.

Forecast: We took the wrong action. Does the update still learn something useful?

We update only the action we took, $a_{1}$ . Its old value is 2. Why this step? Q-learning updates the $(s, a)$ pair actually experienced — here $(s, a_{1})$ .
The target uses $max$ , not the action we took. $target = - 1 + 0.9 \times 6 = 4.4$ . Why this step? This is the off-policy heart of the algorithm: the future term assumes we'll play optimally from $s^{'}$ ( $max$ , i.e. it credits the optimal action $a^{*}$ of the next state), even though right now we're exploring badly. That's why Q-learning learns $Q^{*}$ while misbehaving. (Contrast: SARSA would use the value of the action it actually takes next.)
Nudge. $Q (s, a_{1}) \leftarrow 2 + 0.1 \times (4.4 - 2) = 2 + 0.24 = 2.24$ .

Verify: $2 + 0.1 (4.4 - 2) = 2.24$ . ✓ Cell H: even a suboptimal move produces a correct-toward-optimal update. See Temporal Difference Learning and Exploration-Exploitation Trade-off for why exploring is required for convergence.

Forecast: The true best next value is 0. Guess what $max$ reports.

Take the max of the noisy estimates. $max (1.3, - 1.1) = 1.3$ . Why this step? $max$ always grabs the highest number, and noise pushes some estimate above the true 0. So $max$ systematically reports a value above the true 0 — this is maximization bias.
Feed it into a target. With $r = 0, γ = 0.9$ : $target = 0 + 0.9 \times 1.3 = 1.17$ — but the honest target is $0 + 0.9 \times 0 = 0$ . We over-shot by $1.17$ . Why this step? Repeatedly selecting and evaluating with the same noisy max bakes in a positive error.
The fix — Double Q-learning. Keep two independent tables $Q_{A}$ and $Q_{B}$ , each learned from different samples so their noise is uncorrelated. On each update we flip a coin to decide which table gets updated; the OTHER one supplies the evaluation. Concretely, when updating $Q_{A}$ : first select the greedy action using $Q_{A}$ , $a^{*} = ar g max_{a^{'}} Q_{A} (B, a^{'})$ ; then evaluate that action with $Q_{B}$ , giving target $= r + γ Q_{B} (B, a^{*})$ . In our numbers, if $Q_{A}$ picks $a_{1}$ but $Q_{B} (B, a_{1}) = - 0.4$ , the target becomes $0 + 0.9 \times (- 0.4) = - 0.36$ — no systematic upward bias. Why this step? Independent tables mean the table that picks the winner isn't the same one that rates it, so an estimate that got lucky-high in $Q_{A}$ is rated by an independent $Q_{B}$ that did not share that luck — the bias cancels on average. The diagram below shows this select-in-one / evaluate-in-the-other split.

Reading the figure. Two stacked boxes are the tables $Q_{A}$ (blue) and $Q_{B}$ (green). The blue arrow shows $Q_{A}$ selecting the winning action $a^{*}$ (arg-max), and the green arrow shows that same $a^{*}$ being evaluated by $Q_{B}$ to form the target. The dashed coin at the top decides which table is the "selector" this step; next step the roles swap. The red label marks where ordinary Q-learning would instead evaluate with the same table — the source of the bias.

Verify: biased target $= 0.9 \times 1.3 = 1.17$ ; unbiased-example target $= 0.9 \times (- 0.4) = - 0.36$ . ✓ Cell I demonstrated. Related: Deep Q-Networks (DQN), Prioritized Experience Replay.

Forecast: Translate the story into numbers first, then guess the new Q.

Extract the tuple $(s, a, r, s^{'})$ . $r = 4$ , next-state best value $= 7$ . The messy sentence collapses to four objects. Why this step? Q-learning is domain-blind — grids, elevators, and servers are all just $(s, a, r, s^{'})$ streams.
Build the TD target. $target = r + γ max_{a^{'}} Q (s^{'}, a^{'}) = 4 + 0.95 \times 7 = 4 + 6.65 = 10.65$ . Why this step? Reward now (latency met, $+ 4$ ) plus the discounted promise of a lightly-loaded next state (value 7, shrunk by $γ = 0.95$ ).
Compute the TD error. $δ = target - Q (s, a) = 10.65 - 3.5 = 7.15$ . Why this step? We badly under-rated this assignment — the surprise is large and positive.
Nudge. $Q (s, a) \leftarrow 3.5 + 0.2 \times 7.15 = 3.5 + 1.43 = 4.93$ . Why this step? We move 20% of the way toward the target. The scheduler now rates this assignment higher and will repeat it under similar load. Same math powers Atari Game Playing and Robot Control.

Verify: $3.5 + 0.2 (10.65 - 3.5) = 4.93$ , and it lands strictly between the old value 3.5 and target 10.65 (sanity: units are all "expected discounted reward," consistent). ✓ Cell J: an English paragraph became one arithmetic line.

This example uses a figure — watch value seep backward from the goal.

The figure, read left to right. Four boxes are the states $s_{0}, s_{1}, s_{2}, GOAL$ (green). The white arrows are the corridor moves, each tagged with its reward (yellow): three $r = - 1$ steps and the final $r = + 10$ into the goal. The red row underneath shows every $Q$ value after episode 1 — notice only $s_{2}$ has tasted the goal ( $Q = 10$ ); the earlier states still read $- 1$ . The green row shows the same $Q$ values after episode 2 — now $s_{1}$ has jumped to $8$ because the goal's value has crawled one box to the left. The yellow annotation arrow points at exactly this one-box-per-episode diffusion.

Forecast: After ONE pass, how many of the three Q-values are non-zero, and how many have actually "felt" the +10 goal? (Careful — this surprises people.)

Answer to the forecast: All three become non-zero after one pass, but only one ( $s_{2}$ ) has felt the goal; the other two just recorded the $- 1$ step penalty. The goal's value has not reached $s_{0}$ or $s_{1}$ yet.

Ordering convention: within an episode the agent physically visits $s_{0}$ first, then $s_{1}$ , then $s_{2}$ , then GOAL — so the updates fire in that order. Keep this in mind; it is the whole trick.

Ep 1, update at $s_{0} \to s_{1}$ (fires first). At this moment $max_{a^{'}} Q (s_{1}, \cdot) = 0$ (nothing learned yet). $target = - 1 + 0.9 \times 0 = - 1$ , $δ = - 1 - 0 = - 1$ . With $α = 1$ : $Q (s_{0}, \to) = - 1$ . Why this step? The first update sees only the step penalty; the downstream value is still 0.
Ep 1, update at $s_{1} \to s_{2}$ (fires second). Still $max_{a^{'}} Q (s_{2}, \cdot) = 0$ . $target = - 1 + 0.9 \times 0 = - 1 \Rightarrow Q (s_{1}, \to) = - 1$ . Why this step? $s_{2}$ hasn't touched the goal yet at this instant, so again only $- 1$ registers.
Ep 1, update at $s_{2} \to GOAL$ (fires third). GOAL is terminal so $max = 0$ . $target = 10 + 0.9 \times 0 = 10 \Rightarrow Q (s_{2}, \to) = 10$ . After episode 1: $Q (s_{0}) = - 1, Q (s_{1}) = - 1, Q (s_{2}) = 10$ . Only $s_{2}$ felt the goal — the red row of the figure.
Ep 2, update at $s_{0} \to s_{1}$ (fires first). Now $max_{a^{'}} Q (s_{1}, \cdot) = - 1$ . $target = - 1 + 0.9 \times (- 1) = - 1.9 \Rightarrow Q (s_{0}, \to) = - 1.9$ . Why this step? $s_{0}$ absorbs $s_{1}$ 's (still goal-less) value — it gets slightly worse before it gets better.
Ep 2, update at $s_{1} \to s_{2}$ (fires second). Now $max_{a^{'}} Q (s_{2}, \cdot) = 10$ . $target = - 1 + 0.9 \times 10 = 8 \Rightarrow Q (s_{1}, \to) = 8$ . Why this step? The goal's value has now crawled from $s_{2}$ into $s_{1}$ — one box left, exactly as the figure's yellow arrow shows.
Ep 2, update at $s_{2} \to GOAL$ (fires third). $target = 10 \Rightarrow Q (s_{2}, \to) = 10$ (unchanged). After episode 2: $Q (s_{0}) = - 1.9, Q (s_{1}) = 8, Q (s_{2}) = 10$ — the green row.

Verify: After ep1: $(Q_{s_{0}}, Q_{s_{1}}, Q_{s_{2}}) = (- 1, - 1, 10)$ . After ep2: $Q (s_{0}) = - 1 + 0.9 (- 1) = - 1.9$ , $Q (s_{1}) = - 1 + 0.9 (10) = 8$ . ✓ Cell K: value crawls one state backward per episode. The Bellman Equations fixed point $Q (s_{0}) = - 1 + 0.9 (- 1 + 0.9 (- 1 + 0.9 \cdot 10)) = 6.2$ is only reached after enough passes. This is exactly Markov Decision Processes planning done by trial and error.

Recall

Recall What is the TD error

$δ$ and how does it drive the update? $δ = target - Q (s, a)$ , the "surprise" ::: the whole update is $Q \leftarrow Q + α δ$ ; when $δ = 0$ nothing changes.

Recall When is

$max_{a^{'}} Q (s^{'}, a^{'})$ equal to zero? When $s^{'}$ is terminal ::: because terminal states are seeded with $Q = 0$ , so the future term vanishes (Cell A).

Recall With

$α = 1$ , what does $Q (s, a)$ become after an update? Exactly the TD target ::: the old value is fully discarded (Cell F).

Recall In

$ε$ -greedy, what does $ε$ control? The probability of picking a random (exploratory) action ::: with probability $1 - ε$ the agent picks the greedy optimal action $a^{*}$ .

Recall Why does Q-learning learn the optimal policy even while exploring badly?

The target uses $max_{a^{'}} Q (s^{'}, a^{'})$ ::: it evaluates the best future action $a^{*}$ , not the one actually taken — this is what "off-policy" means (Cell H).

Recall What causes maximization bias and what fixes it?

Taking $max$ over noisy estimates over-values states ::: Double Q-learning fixes it by using separate tables to select and to evaluate (Cell I).

Recall In a 3-state corridor, how far does goal-value propagate per episode?

One state backward per episode ::: value diffuses one hop at a time (Cell K).