Exercises — Discount factor and returns

What we do: apply the definition $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$ with $t=3$.

The reward that follows the action at $t=3$ is $R_{t+1}=R_4$ — not $R_3$. Plugging $k=0,1,2$: $G_3=R_4+\gamma R_5+{\gamma }^2{R}_6+\cdots$ Why: the action at time $t$ causes the reward $R_{t+1}$; the reward you already collected getting to $t$ is history and never appears in $G_t$.

What we do: compare each $\gamma$ against the endpoints $0$ and $1$ (recall the allowed range $0\le \gamma <1$).

• (a) $\gamma =0$ → myopic. Every ${\gamma }^k$ with $k\ge 1$ is $0$, so only $R_{t+1}$ survives.
• (b) $\gamma =0.999$ → far-sighted. Weights decay only $0.1\%$ per step, so rewards ~1000 steps out still count.
• (c) $\gamma =0.5$ → myopic-ish (short-sighted). Weight halves each step; by step 4 it's already $1/16$.

What we do: factor the constant out; it becomes a pure geometric series (allowed since $\gamma =0.8<1$). $G_t={\sum }_{k=0}^{\infty }{\gamma }^k\cdot 2=2{\sum }_{k=0}^{\infty }0.8^k=2\cdot \frac{1}{1-0.8}=2\cdot 5=10.$ Why: constant reward $R$ gives $R\cdot \frac{1}{1-\gamma }$. Notice an infinite stream of $+2$'s is worth only $10$ — discounting tames infinity into a finite number.

What we do: the return is always the infinite sum, but here every reward from $R_{t+5}$ onward is $0$, so the infinite tail contributes nothing and truncation is exact: $G_t=\underset{\text{first four terms}}{\underbrace{4+0.5(-2)+0.5^2(6)+0.5^3(0)}}+\underset{=0\text{(a terminated episode)}}{\underbrace{0.5^4\cdot 0+0.5^5\cdot 0+\cdots }}.$ The zero-tail adds nothing, so we keep only the four non-zero terms: $G_t=4-1+0.25\cdot 6+0=4-1+1.5=\mathrm{4.5.}$ Why: the exponent counts steps into the future: $R_{t+1}$ gets ${\gamma }^0=1$, $R_{t+2}$ gets ${\gamma }^1$, and so on. The reward $6$ that arrives two steps late is worth only $1.5$ now. The infinite sum notation is still correct — a terminating episode is just an infinite tail of zeros.

What we do: start from the end (where $G=0$) and fold inward. This is exactly how TD / dynamic programming propagate values.

• $G_{t+3}=R_{t+4}+0.5G_{t+4}=0+0.5\cdot 0=0$
• $G_{t+2}=R_{t+3}+0.5G_{t+3}=6+0.5\cdot 0=6$
• $G_{t+1}=R_{t+2}+0.5G_{t+2}=-2+0.5\cdot 6=1$
• $G_t=R_{t+1}+0.5G_{t+1}=4+0.5\cdot 1=4.5$ ✓ Why: the recursion needs a terminal $G=0$ to launch — without it there is no place to start folding. The answer matches 2.2, as it must.

What we do: apply the definition. With $\gamma =1$ every weight ${\gamma }^k=1$, so the return is the plain undiscounted sum of the four rewards: $G_t=3+1+4+2=10.$ This is finite and well-defined because the episode stops after $4$ terms — the infinite tail is all zeros, so $\gamma =1$ causes no divergence. Why it's safe: the danger of $\gamma =1$ is only an infinite number of non-zero terms; a guaranteed-terminating episode has finitely many, so no blow-up. With $\gamma =0.5$: $G_t=3+0.5(1)+0.5^2(4)+0.5^3(2)=3+0.5+1+0.25=\mathrm{4.75.}$ Comment: discounting ($\gamma =0.5$) shrinks the return from $10$ to $4.75$ by fading later rewards; $\gamma =1$ treats all four rewards as equally valuable. On a terminating episode both are legitimate — $\gamma =1$ is the "no time-preference" choice.

What we do: use $|G_t|\le \frac{R_{\max }}{1-\gamma }$ with $R_{\max }=5$ (recall $R_{\max }$ = largest reward magnitude). $|G_t|\le \frac{5}{1-0.9}=\frac{5}{0.1}=50.$ As $\gamma \to 1$: $\frac{5}{1-\gamma }\to \infty$. Why: the denominator $1-\gamma \to 0$, so the bound explodes — long horizons (large $\gamma$) can let returns grow without limit in continuing tasks (tasks that never terminate). This is the mathematical reason we usually keep $\gamma <1$.

What we do: plug $\gamma =0.9$ into both formulas. $\underset{\text{weight scale}}{\underbrace{\frac{1}{1-\gamma }}}=\frac{1}{0.1}=10,\underset{\text{mean step}}{\underbrace{\mathbb{E}[k]=\frac{0.9}{0.1}}}=9.$ The mean step $\mathbb{E}[k]=9$ is the honest average look-ahead: the average of $k$ under the normalized weights $P(k)=(1-\gamma ){\gamma }^k$ (defined just above). People quote the weight scale $\frac{1}{1-\gamma }=10$ because it's the total (un-normalized) weight $\sum {\gamma }^k$ and is one larger; the two nearly coincide as $\gamma \to 1$. Why: the mean sits one below the scale because $\mathbb{E}[k]=\frac{1}{1-\gamma }-1$.

What we do: constant-reward formula $G_t=\frac{R}{1-\gamma }$.

• Future A ($R=+1$): $\gamma =0.5\Rightarrow \frac{1}{0.5}=2$; $\gamma =0.99\Rightarrow \frac{1}{0.01}=100$. Raising $\gamma$ increases it.
• Future B ($R=-1$): $\gamma =0.5\Rightarrow \frac{-1}{0.5}=-2$; $\gamma =0.99\Rightarrow \frac{-1}{0.01}=-100$. Raising $\gamma$ decreases it (more negative). Why: $\gamma$ amplifies whatever the future is. For positive futures bigger $\gamma$ helps; for penalty-laden futures it hurts. "Bigger $\gamma$ = bigger return" is only true when the future is positive.

What we do: invert $G_t=\frac{R}{1-\gamma }$. $30=\frac{3}{1-\gamma }\Rightarrow 1-\gamma =\frac{3}{30}=0.1\Rightarrow \gamma =\mathrm{0.9.}$ Why: the return-to-reward ratio $\frac{G_t}{R}=\frac{1}{1-\gamma }=10$ directly reveals the horizon scale, and $\gamma =1-\frac{1}{10}=0.9$ (safely inside $0\le \gamma <1$).

What we do: substitute the reward into the return and recognise a geometric series with ratio $\gamma \rho$. $G_t={\sum }_{k=0}^{\infty }{\gamma }^k(c{\rho }^k)=c{\sum }_{k=0}^{\infty }(\gamma \rho {)}^k=\frac{c}{1-\gamma \rho }.$ Why: two multiplicative factors per step ($\gamma$ from discounting, $\rho$ from reward growth) merge into a single ratio $\gamma \rho$; convergence needs $|\gamma \rho |<1$. Numbers: $\gamma \rho =0.9\times 1.05=0.945$, so $G_t=\frac{1}{1-0.945}=\frac{1}{0.055}\approx \mathrm{18.18.}$ Sanity: $\rho >1$ makes rewards grow, so $G_t\approx 18.18$ exceeds the constant-reward value $\frac{1}{1-0.9}=10$. Good.

What we do: apply the definition, four non-zero terms then a zero-tail (episode terminates). $G_t=-1-0.9-0.9^2+0.9^3(100).$ Compute: $-1-0.9-0.81=-2.71$; and $0.9^3=0.729$, so $0.729\times 100=72.9$. $G_t=-2.71+72.9=70.19>0.$ Why: even though the jackpot is discounted to $72.9$ (down from $100$), it dwarfs the small early penalties. Entering is worth it. The lesson: delay shrinks a reward but a big enough reward still dominates.

What we do: split off the $k=0$ term, then factor $\gamma$ from the rest. $G_t=\underset{k=0}{\underbrace{{\gamma }^0{R}_{t+1}}}+{\sum }_{k=1}^{\infty }{\gamma }^k{R}_{t+k+1}.$ Re-index the tail with $j=k-1$ (so $k=j+1$, and $j$ runs $0\to \infty$): ${\sum }_{k=1}^{\infty }{\gamma }^k{R}_{t+k+1}={\sum }_{j=0}^{\infty }{\gamma }^{j+1}{R}_{t+j+2}=\gamma {\sum }_{j=0}^{\infty }{\gamma }^j{R}_{(t+1)+j+1}=\gamma G_{t+1}.$ Therefore $G_t=R_{t+1}+\gamma G_{t+1}$. $■$ Why: the tail is exactly the return starting one step later, scaled by one factor of $\gamma$. This self-reference is the seed of every Bellman equation.

Step 1 — valid distribution. ${\sum }_{k=0}^{\infty }(1-\gamma ){\gamma }^k=(1-\gamma )\cdot \frac{1}{1-\gamma }=1.$ ✓ (uses the Geometric series sum, valid because $\gamma <1$; the $(1-\gamma )$ is exactly the normalizer built in Exercise 3.2). Step 2 — the mean. $\mathbb{E}[k]={\sum }_{k=0}^{\infty }k(1-\gamma ){\gamma }^k=(1-\gamma ){\sum }_{k=0}^{\infty }k{\gamma }^k=(1-\gamma )\cdot \frac{\gamma }{(1-\gamma {)}^2}=\frac{\gamma }{1-\gamma }.$ Why: we used the standard identity ${\sum }_{k}k{\gamma }^k=\frac{\gamma }{(1-\gamma {)}^2}$; the normalizer $(1-\gamma )$ cancels one power of $(1-\gamma )$, leaving $\frac{\gamma }{1-\gamma }$ — one less than the scale $\frac{1}{1-\gamma }$.

The danger of $\gamma =1$: in a continuing task (one with no terminal state) the sum has infinitely many terms; with $\gamma =1$ the bound $\frac{R_{\max }}{1-\gamma }=\frac{10}{0}$ is undefined and $G_t$ can diverge to $\infty$. $\gamma =1$ is only safe when termination is guaranteed (an episodic task, as in Exercise 2.4). This is why the allowed range on this page is $0\le \gamma <1$. Choosing $\gamma$: require $\frac{R_{\max }}{1-\gamma }\le 200$, i.e. $\frac{10}{1-\gamma }\le 200$. $1-\gamma \ge \frac{10}{200}=0.05\Rightarrow \gamma \le \mathrm{0.95.}$ So any $\gamma \le 0.95$ (e.g. $\gamma =0.95$) caps $|G_t|\le 200$. Why: the bound $\frac{R_{\max }}{1-\gamma }$ is monotone in $\gamma$; solving the inequality gives the ceiling on $\gamma$ that keeps returns tame. Turning the maximisation of $G_t$ into a well-posed objective is exactly what the Reward hypothesis (all goals = maximising expected cumulative reward) needs.

$G_t=\sum {\gamma }^k{R}_{t+k+1}$ (return) ::: total discounted future reward, with $0\le \gamma <1$. $G_t=R_{t+1}+\gamma G_{t+1}$ ::: recursion, needs terminal $G=0$ to unfold. $\sum {\gamma }^k=\frac{1}{1-\gamma }$ ::: weight scale (total of all discount weights) / horizon. $\mathbb{E}[k]=\frac{\gamma }{1-\gamma }$ ::: true mean look-ahead (one less than the scale). $|G_t|\le \frac{R_{\max }}{1-\gamma }$ ::: return is finite for $\gamma <1$. $\gamma =1$ on a terminating episode ::: legal — the plain undiscounted sum of finitely many rewards.