Discount factor and returns: Worked examples — AI-ML deep dive

This is the "roll-up-your-sleeves" companion to the parent note on returns and $\gamma$. Over there we built the machinery: the return $G_{t} = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1}$ , the discount factor $γ \in [0, 1]$ , the bound $\frac{R _{m a x}}{1 - γ}$ , and the recursion $G_{t} = R_{t + 1} + γ G_{t + 1}$ . Here we do only one thing: hit every kind of scenario the topic can throw at you, one worked example at a time.

Before line one: a quick reminder of the three symbols we will use constantly.

Recall The three symbols, in plain words

$R_{t + 1}$ ::: the reward you receive one step after being at time $t$ (the action at $t$ pays out at $t + 1$ ).
$γ$ ::: the "trust dial on the future", a number between $0$ and $1$ ; every extra step into the future multiplies its reward by another $γ$ .
$G_{t}$ ::: the return — add up all future rewards, each shrunk by its own power of $γ$ .

The scenario matrix

Every problem about returns is really one of the cells below. If you can do one example per row, you can do them all. (Prereq threads: Geometric series, Episodic vs continuing tasks, and the recursion feeds Value functions and Bellman equations.)

#	Cell class	What makes it different	Covered by
A	Constant reward, continuing task	pure geometric series, infinite sum	Ex 1
B	Constant reward, negative sign	tests "bigger $γ$ ≠ bigger return"	Ex 2
C	Finite episode, mixed signs	terminates → sum stops	Ex 3
D	Same episode via the recursion	needs terminal condition $G = 0$	Ex 4
E	Degenerate $γ = 0$ (myopic)	only $R_{t + 1}$ survives	Ex 5
F	Limiting $γ \to 1$ (episodic OK)	undiscounted finite sum	Ex 6
G	Effective horizon / mean step	$\frac{1}{1 - γ}$ vs $\frac{γ}{1 - γ}$	Ex 7
H	Delayed lump sum (word problem)	one reward far in the future	Ex 8
I	Periodic reward (exam twist)	reward every $m$ steps → geometric in $γ^{m}$	Ex 9
J	Shifting $γ$ changes the decision	two paths, which wins depends on $γ$	Ex 10

The figures below carry the geometry: how the discounted weights stack up (Ex 1), and how the choice between two paths flips as you turn the $γ$ dial (Ex 10).

The worked examples

Worked example Example 2 — Cell B: constant

negative reward Reward $R = - 2$ every step. Compute $G_{t}$ for $γ = 0.5$ and for $γ = 0.9$ , then say which $γ$ gives the larger return.

Forecast: the classic trap — does bigger $γ$ give the bigger number?

Same factoring as Ex 1, now with $- 2$ . $G_{t} = - 2 \cdot \frac{1}{1 - γ}$ Why this step? Constant reward → geometric series again; only the sign of the constant changed.
Plug both values. $γ = 0.5 : G_{t} = - 2 \cdot \frac{1}{0.5} = - 4, γ = 0.9 : G_{t} = - 2 \cdot \frac{1}{0.1} = - 20$ Why this step? Larger $γ$ sums more of the future, and the future here is all penalties — so it makes the return more negative, i.e. smaller.

Verify: $- 4 > - 20$ , so the smaller $γ$ gives the larger return. This is precisely the "bigger $γ$ always wins" mistake, refuted with numbers. $γ$ amplifies whatever the future is.

Worked example Example 3 — Cell C: finite episode, mixed signs

An episode ends after three rewards: $R_{t + 1}, R_{t + 2}, R_{t + 3} = 4, - 2, 6$ , all later rewards $0$ . Use $γ = 0.5$ . Find $G_{t}$ .

Forecast: the raw sum $4 - 2 + 6 = 8$ — will discounting push it above or below $8$ ?

Apply the definition term by term, stopping where rewards become $0$ . $G_{t} = 4 + γ (- 2) + γ^{2} (6)$ Why this step? An episode that terminates means every $R$ after $R_{t + 3}$ is $0$ , so the infinite sum truncates to three live terms.
Substitute $γ = 0.5$ . $G_{t} = 4 + 0.5 (- 2) + 0.25 (6) = 4 - 1 + 1.5 = 4.5$ Why this step? Each step multiplies by an extra $γ$ ; the good reward $6$ arrives late so it is shrunk to $1.5$ .

Verify: $4.5 < 8$ because the biggest reward ( $6$ ) is the latest, so it got the heaviest discount. Sanity: with $γ = 1$ we'd get exactly $8$ (checked in Ex 6-style below).

Worked example Example 4 — Cell D: the same episode, via the recursion

Same rewards $4, - 2, 6$ , $γ = 0.5$ , episode ends after $R_{t + 3}$ so $G_{t + 3} = 0$ . Compute $G_{t}$ using $G_{t} = R_{t + 1} + γ G_{t + 1}$ , working backwards.

Forecast: it must match Ex 3's $4.5$ — if it doesn't, you dropped the terminal condition.

Start at the terminal condition. $G_{t + 3} = 0$ Why this step? Nothing comes after the last reward; the recursion needs this anchor to start.
Step back one. $G_{t + 2} = R_{t + 3} + γ G_{t + 3} = 6 + 0.5 \cdot 0 = 6$ Why this step? The recursion $G = R_{next} + γ G_{after}$ applied at $t + 2$ : its next reward is $R_{t + 3} = 6$ and the return after it is the terminal $G_{t + 3} = 0$ , so $G_{t + 2}$ is just the last reward.
Step back again. $G_{t + 1} = R_{t + 2} + γ G_{t + 2} = - 2 + 0.5 \cdot 6 = 1$ Why this step? Same recursion at $t + 1$ : its next reward is $R_{t + 2} = - 2$ and the future return is the $G_{t + 2} = 6$ we just computed — we reuse it instead of re-summing, which is the whole point of the recursion.
Final step. $G_{t} = R_{t + 1} + γ G_{t + 1} = 4 + 0.5 \cdot 1 = 4.5$ Why this step? One more application at $t$ : next reward $R_{t + 1} = 4$ plus $γ$ times the already-known $G_{t + 1} = 1$ . This backward chaining is exactly how dynamic programming and TD learning build values.

Verify: $4.5$ matches Ex 3 exactly. The backward recursion and the forward sum agree.

Worked example Example 8 — Cell H: a delayed lump sum (word problem)

A robot vacuum gets $0$ reward every step, except it earns $+ 50$ exactly $7$ steps from now (i.e. $R_{t + 7} = 50$ ), then the episode ends. With $γ = 0.9$ , what is that promised reward worth now as part of $G_{t}$ ?

Forecast: $50$ , but haircut by how much?

Identify which power of $γ$ multiplies this single reward. $G_{t} = γ^{6} \cdot 50$ Why this step? In $G_{t} = \sum_{k} γ^{k} R_{t + k + 1}$ , the reward $R_{t + 7}$ has $t + k + 1 = t + 7 \Rightarrow k = 6$ , so it carries $γ^{6}$ .
Compute. $G_{t} = 0. 9^{6} \cdot 50 = 0.531441 \cdot 50 = 26.57205$ Why this step? Only one live reward, so the return is just its discounted value — every $0$ contributes nothing.

Verify: the future $+ 50$ is worth about $\$ 26.57 $t o d a y — r o ug h l y ha l v e d b y se v e n s t e p so f$ 0.9 $d i sco u n t in g, co n s i s t e n tw i t h "$ \gamma $a s anin t er es t r a t e / s u r v i v a l p r o babi l i t y ". O f f - b y - o n ec h ec k : i t i s$ \gamma^6 $, * * n o t * *$ \gamma^7 $, b ec a u se$ R_{t+1} $i s t h e * f i r s t * f u t u r er e w a r d (p o w er$ 0$).

Worked example Example 9 — Cell I: periodic reward (exam twist)

An agent earns $+ 1$ on steps $t + 2, t + 4, t + 6, \dots$ (every second step, starting at $t + 2$ ) and $0$ otherwise, forever, with $γ = 0.5$ . Find $G_{t}$ .

Forecast: it's a geometric series in disguise — in $γ$ or in $γ^{2}$ ?

Write only the live terms. $G_{t} = γ^{1} \cdot 1 + γ^{3} \cdot 1 + γ^{5} \cdot 1 + \dots$ Why this step? Reward $R_{t + 2}$ has $k = 1$ , $R_{t + 4}$ has $k = 3$ , and so on — odd powers only.
Factor out $γ$ and recognise a geometric series with ratio $γ^{2}$ . $G_{t} = γ (1 + γ^{2} + γ^{4} + \dots) = γ \cdot \frac{1}{1 - γ ^{2}}$ Why this step? The gap between rewards is $2$ steps, so each successive live term is $γ^{2}$ times the previous — the common ratio is $γ^{2}$ , not $γ$ .
Plug $γ = 0.5$ . $G_{t} = \frac{0.5}{1 - 0.25} = \frac{0.5}{0.75} = \frac{2}{3} \approx 0.6667$ Why this step? Direct substitution into the closed form; $γ^{2} = 0.25$ .

Verify: partial sum $0.5 + 0.125 + 0.03125 + \dots$ climbs toward $0.6667$ . Cross-check: full-rate ( $+ 1$ every step from $t + 2$ ) would be $γ \cdot \frac{1}{1 - γ} = 0.5 \cdot 2 = 1$ ; halving the reward frequency gives less, and $0.667 < 1$ . ✓

Worked example Example 10 — Cell J: the

$γ$ dial flips the decision Two policies from the same state. Path P pays $R_{t + 1} = 5$ then $0$ forever (grab-now). Path Q pays $0, 0, 0$ then $+ 12$ at $R_{t + 4}$ then $0$ forever (patient payoff). For which $γ$ does P beat Q, and where does the decision flip?

Forecast: small $γ$ favours which one? Guess the crossover $γ$ .

Write both returns. $G^{P} = 5, G^{Q} = γ^{3} \cdot 12$ Why this step? P's only reward is at $k = 0$ (power $γ^{0} = 1$ ); Q's only reward $R_{t + 4}$ sits at $k = 3$ (power $γ^{3}$ ).
Set them equal to find the crossover. $5 = 12 γ^{3} \Rightarrow γ^{3} = \frac{5}{12} \Rightarrow γ = (\frac{5}{12})^{1/3} \approx 0.7472$ Why this step? At the crossover the two returns tie; below it P wins, above it Q wins, because $G^{Q}$ grows with $γ$ while $G^{P}$ is flat.
Read off the cases.

$γ < 0.7472$ (impatient) → $G^{P} > G^{Q}$ , grab now.
$γ > 0.7472$ (patient) → $G^{Q} > G^{P}$ , wait for the big payoff.
$γ = 0.7472$ → tie.

Verify (spot-check): at $γ = 0.5$ , $G^{Q} = 0.125 \cdot 12 = 1.5 < 5$ → P wins ✓. At $γ = 0.9$ , $G^{Q} = 0.729 \cdot 12 = 8.748 > 5$ → Q wins ✓. The dial genuinely changes the optimal action — see the two curves crossing in the figure.

Recap

Recall One-line summary of every cell

Constant reward → $\frac{R}{1 - γ}$ (Ex 1–2); finite episode → truncated term-by-term sum, matched by the backward recursion with $G_{terminal} = 0$ (Ex 3–4); $γ = 0$ myopic keeps only $R_{t + 1}$ (Ex 5); $γ = 1$ is safe only when episodic (Ex 6); horizon scale $\frac{1}{1 - γ}$ vs mean step $\frac{γ}{1 - γ}$ differ by one (Ex 7); a single delayed reward is worth $γ^{k} R$ (Ex 8); periodic reward → geometric in $γ^{m}$ (Ex 9); and $γ$ can flip which policy is optimal (Ex 10).

Return to the parent note · related: Reward hypothesis, Markov Decision Process.