Visual walkthrough — Discount factor and returns

We are going to answer one question, slowly and visually:

"An agent will collect rewards $R_{t+1},R_{t+2},R_{t+3},\dots$ forever. How do we turn that endless list into one number that says how good the future is — and why doesn't that number explode to infinity?"

---

Step 1 — Draw the reward timeline

WHAT. Picture a horizontal track. The agent sits at time step $t$. It acts, and one step later a reward lands. The very first reward is written $R_{t+1}$ — the little subscript $t+1$ just means "the reward that arrives after the action at time $t$". The next is $R_{t+2}$, then $R_{t+3}$, and so on with no end.

WHY. Before we add anything up, we need to see the objects we are adding. Each reward is a box sitting on the timeline. The whole page is about combining these boxes into one number.

PICTURE. Look at the amber boxes below — they march off to the right forever.

---

Step 2 — Why we can't just add them all

WHAT. Suppose we tried the naive thing: total future reward $=R_{t+1}+R_{t+2}+R_{t+3}+\cdots$. If each reward is, say, $+1$ and there are infinitely many, the sum grows past any number you name.

WHY. A quantity that equals $\infty$ is useless for comparing plans — every plan scores infinity, so no plan is better than another. We need a rule that makes the pile finite without throwing information away. That rule is discounting.

PICTURE. The staircase below climbs forever — that is the naive sum, and it never levels off.

---

Step 3 — Introduce the shrink factor $\gamma$

WHAT. We attach a number $\gamma$ (Greek "gamma"), chosen between $0$ and $1$, to the future. The reward $k$ steps out is multiplied by $\gamma$ used $k$ times, i.e. by ${\gamma }^k$:

$$\underset{\times {\gamma }^0=1}{\underbrace{R_{t+1}}}+\underset{\times {\gamma }^1}{\underbrace{\gamma R_{t+2}}}+\underset{\times {\gamma }^2}{\underbrace{{\gamma }^2{R}_{t+3}}}+\cdots$$

Read the annotations right under each term: the first reward is untouched (${\gamma }^0=1$), the next is shrunk once, the next shrunk twice, and so on.

WHY ${\gamma }^k$ and not some other shrink? Because we want every extra step of waiting to cost the same fraction. "Wait one more step ⇒ multiply by $\gamma$ once more" is the simplest fair rule, and — crucially — it makes the sum a geometric series, the one infinite sum we know how to collapse (see Geometric series). Any other decay would break that clean trick.

PICTURE. The boxes now shrink as you go right; the tall infinite staircase becomes a curve that flattens.

---

Step 4 — Sum the weights: the geometric-series trick

WHAT. Assume for a moment every reward equals $1$. Then the return is just the sum of the weights — call that sum $S$. In plain words, $S$ is the total of all the shrink factors ${\gamma }^0,{\gamma }^1,{\gamma }^2,\dots$ added together: $S={\sum }_{k=0}^{\infty }{\gamma }^k=1+\gamma +{\gamma }^2+{\gamma }^3+\cdots$ Now that $S$ has a name and a meaning, multiply the whole line by $\gamma$ and line the two rows up:

$$\begin{array}{lcl}S& =& 1+\gamma +{\gamma }^2+{\gamma }^3+\cdots \\ \gamma S& =& \gamma +{\gamma }^2+{\gamma }^3+\cdots \end{array}$$

Subtract the second row from the first. Everything in the tail cancels and only the lonely $1$ survives: $S-\gamma S=1⟹S(1-\gamma )=1⟹S=\frac{1}{1-\gamma }.$

WHY this move? Shifting by one step and subtracting is the only algebra where an infinite tail annihilates itself. It works only because $\gamma <1$, which forces ${\gamma }^k\to 0$ so the far tail truly vanishes.

PICTURE. The figure shows the two shifted staircases; the shaded overlap is the part that cancels, leaving a single block of height $1$ over a base of $(1-\gamma )$.

---

Step 5 — Bound a real return with worst-case rewards

WHAT. Real rewards aren't all $1$; they wobble. Suppose they never exceed $R_{\max }$ in size, i.e. $|R|\le R_{\max }$. Then

$$|G_t|=\left|\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}\right|\le \sum _{k=0}^{\infty }{\gamma }^k\underset{\le R_{\max }}{\underbrace{|R_{t+k+1}|}}\le R_{\max }\sum _{k=0}^{\infty }{\gamma }^k=\frac{R_{\max }}{1-\gamma }.$$

Read left to right: first the absolute value slides inside the sum. The rule that lets us do this is the triangle inequality: the size of a sum is never bigger than the sum of the sizes, $\left|\sum a_k\right|\le \sum |a_k|$ (adding numbers with mixed signs can only make things cancel, never grow larger than adding their magnitudes). Since each ${\gamma }^k$ is positive, $|{\gamma }^{k}R|={\gamma }^k|R|$, so the ${\gamma }^k$ stay outside. Then each $|R|$ is pushed up to its worst case $R_{\max }$, and finally Step 4 collapses the weights.

WHY. This is the payoff of the whole page: no matter how nasty the reward stream, the return is trapped below $\frac{R_{\max }}{1-\gamma }$. A trapped number is a finite number — the problem from Step 2 is cured.

PICTURE. The jagged real rewards are all capped by a flat $R_{\max }$ ceiling; the discounted-and-capped area is finite.

---

Step 6 — Peel one box off: the recursive form

WHAT. Take the definition and factor a single $\gamma$ out of everything past the first term:

$$G_t=\underset{\text{first box}}{\underbrace{R_{t+1}}}+\gamma \underset{=G_{t+1}}{\underbrace{(R_{t+2}+\gamma R_{t+3}+\cdots )}}=R_{t+1}+\gamma G_{t+1}.$$

The bracket is exactly the return starting one step later — that's the definition of $G_{t+1}$. So the return talks about itself.

WHY. This turns an infinite sum into a two-piece rule: "immediate reward, plus a discounted copy of the same problem shifted forward." That self-reference is the seed of every Bellman equation — see Value functions and Bellman equations and Temporal Difference learning.

PICTURE. The first amber box is snipped off; everything to its right is boxed as "the same problem, one step later, shrunk by $\gamma$".

---

Step 7 — Every edge case, drawn

WHAT. The formula must survive extremes. Four dials:

• $\gamma =0$ → myopic. Every ${\gamma }^k$ for $k\ge 1$ is $0$; only $R_{t+1}$ survives. $G_t=R_{t+1}$.
• $\gamma \to 1^{-}$ → far-sighted. Weights barely shrink; the ceiling $\frac{R_{\max }}{1-\gamma }$ blows up. Safe only in guaranteed-to-end (episodic) tasks.
• Negative future rewards. $\gamma$ never changes a reward's sign — it only scales its magnitude. So if the future holds penalties, each penalty is shrunk toward zero, but it stays negative; a bigger $\gamma$ shrinks it less, so more of the penalty survives and the return $G_t$ becomes more negative, not larger. $\gamma$ scales the size of what's there, good or bad.
• Terminal step / episode ends. Say the last reward arrives at step $T$. After it there is nothing to collect, so the return from that final step onward is $G_T=0$. That $G_T=0$ is the seed value the recursion in Step 6 needs to start counting backwards.

WHY. A reader who meets any of these in the wild must already have seen it. Each corner behaves differently, so each gets drawn.

PICTURE. Four mini-panels: flat-after-first ($\gamma =0$), barely-decaying ($\gamma \to 1$), a stream with negative boxes, and a stream that stops at a wall.

---

Step 8 — Two worked returns, verified visually

WHAT. Plug numbers in.

WHY. One example exercises the closed form (Step 4), the other exercises the recursion (Step 6) with the terminal zero (Step 7). Getting the same $1.75$ two ways is the whole derivation checking itself.

PICTURE. Left: the $\gamma =0.9$ discounted boxes summing to the dashed line at $10$. Right: the three boxes $2,0,-1$ and their discounted heights adding to $1.75$.

---

The one-picture summary

Everything above, compressed: an infinite row of rewards, each shrunk by one more factor of $\gamma$, whose total area is provably capped by $\frac{R_{\max }}{1-\gamma }$ — and which folds into itself as $G_t=R_{t+1}+\gamma G_{t+1}$.

Recall Feynman retelling — say it out loud

The agent stands at "now" and stares at an endless line of reward boxes stretching into the future. If it valued them all equally the line would be infinitely tall and useless. So it distrusts distance: each step further off, it multiplies the box by $\gamma$, a number a bit under $1$. Far boxes shrink toward nothing. Because the shrink is the same fraction each step, the whole infinite line is a geometric series — shift it by one, subtract, and the endless tail eats itself, leaving $\frac{1}{1-\gamma }$. Cap every reward at $R_{\max }$ and the total is trapped under $\frac{R_{\max }}{1-\gamma }$: finite. Finally, snip off the first box and notice the rest is the same picture, one step later, shrunk once — so $G_t=R_{t+1}+\gamma G_{t+1}$, the recursion that grows up into every Bellman equation.

Recall Quick self-check

Why must $\gamma <1$ for the tail to cancel? ::: Because ${\gamma }^k\to 0$ only when $\gamma <1$; otherwise the far tail never vanishes and the subtraction leaves an infinite leftover. What is $G_t$ if $\gamma =0$? ::: Just $R_{t+1}$ — every later term is multiplied by $0$. What terminal value does the recursion need to start? ::: $G_T=0$ after the final reward.

Related: Markov Decision Process · Reward hypothesis · Geometric series · 5.1.06 Discount factor and returns (Hinglish)