Foundations — Discount factor and returns

Before you can read the parent note fluently, you need to own every symbol it throws at you. Below, each piece is built from zero: plain words → the picture → why the topic needs it. Read top to bottom; each rung stands on the one below it.

1. Time steps: the ticking clock $t$

The symbol $t$ is just "which tick are we talking about." When we write $t+1$, we mean the very next tick. When we write $t+k$, we mean $k$ ticks after $t$ — here $k$ is just a counter for "how many ticks ahead," and we will lean on it heavily soon.

Look at the number line above: the black dots are the ticks. The red dot is "now" ($t$); everything to its right is the future the agent still has to live through. This picture is the stage on which every other symbol performs.

2. The action $A_t$: what the agent does on each tick

Why introduce a symbol for it? Because the timing of everything else hangs on the action. The agent acts ($A_t$), and only then does the environment respond with a reward. Without naming $A_t$ we could not say "the reward is a consequence of the action." Actions and their rules live inside a Markov Decision Process; here we only need to know that a choice happens on every tick.

3. Reward $R_{t+1}$: the score after acting

The subscript matters enormously. Read $R_{t+1}$ as "the reward that shows up on tick $t+1$." It is the consequence of the action $A_t$ you took at tick $t$.

In the figure, each arrow points from an action $A_t$ to the reward it produces. Notice every arrow lands one tick later. That staggering is the entire reason the return starts at $R_{t+1}$.

4. The counter $k$ and powers ${\gamma }^k$

First, pin down the little letter $k$, because it is about to ride on an exponent.

Now the powers:

Why does this appear? Because the discount is applied once per tick into the future. One tick away → one factor of the fade. Two ticks away → two factors, i.e. ${\gamma }^2$. The exponent $k$ — the same counter we just defined — literally counts "how many ticks into the future is this reward."

5. The discount factor $\gamma$: a dial between 0 and 1

Picture $\gamma$ as a dimmer switch on the future:

• $\gamma =0$ → the future is switched fully off. Only $R_{t+1}$ counts. We call this myopic (short-sighted).
• $\gamma \to 1$ → the future is switched fully on. Distant rewards count almost as much as near ones. We call this far-sighted.

The figure plots ${\gamma }^k$ against $k$ for three settings. The red curve ($\gamma =0.9$) fades slowly — the far future still matters. The steep black curve ($\gamma =0.5$) collapses almost immediately — that agent barely sees past the next couple of ticks. Every reward gets multiplied by the height of the curve at its tick.

6. The summation sign $\sum$: adding up a whole stream

You already know $k$ (section 4) — here it is the engine of the sum, stepping through every reward in turn. Unrolled, the expression is exactly: $\underset{k=0}{\underbrace{{\gamma }^0{R}_{t+1}}}+\underset{k=1}{\underbrace{{\gamma }^1{R}_{t+2}}}+\underset{k=2}{\underbrace{{\gamma }^2{R}_{t+3}}}+\cdots$

Why the sigma and not just "$\dots$"? Because we need a precise rule for infinitely many terms. The dots are friendly but vague; $\sum$ tells you exactly which reward gets which power.

7. The return $G_t$: the one number we actually care about

Now every symbol in that line is yours: $t$ is now, $R_{t+1}$ is the next reward, $\gamma$ is the fade dial, $k$ counts ticks ahead, ${\gamma }^k$ applies the fade once per tick, and $\sum$ adds the whole stream. $G_t$ is the single number that answers "how good is the future from here?" — which is precisely what the agent tries to maximise, and what Value functions and Bellman equations average over.

8. Infinity and convergence: does the sum even finish?

Before the guarantee, two pieces of notation:

If you add infinitely many numbers, the total might blow up to $\infty$. So we need a guarantee. The parent note proves that when every reward satisfies $|R_{t+k+1}|\le R_{\max }$ and $\gamma <1$, the return is finite: $|G_t|\le \frac{R_{\max }}{1-\gamma }.$

The engine behind that proof is the Geometric series — a sum where each term is a fixed multiple of the last. Its key fact: $1+\gamma +{\gamma }^2+\cdots =\frac{1}{1-\gamma }\text{when }0\le \gamma <1.$

The Episodic vs continuing tasks distinction is exactly why this matters: episodes that always end have finitely many rewards, so even $\gamma =1$ gives a finite sum with nothing to prove. But continuing tasks run forever and rely on $\gamma <1$ to stay finite. That is the whole reason the field usually insists on strict discounting.

9. The recursion $G_t=R_{t+1}+\gamma G_{t+1}$

Peel the first reward off the front and the rest is the same kind of object, one tick later: $G_t=R_{t+1}+\gamma \left(\underset{=G_{t+1}}{\underbrace{R_{t+2}+\gamma R_{t+3}+\cdots }}\right)=R_{t+1}+\gamma G_{t+1}.$

This self-referential equation (the return defined in terms of the next return) is the seed of every Value functions and Bellman equations relation and of Temporal Difference learning. You need to be comfortable that a quantity can be defined in terms of a shifted copy of itself.

How these foundations feed the topic

The diagram below is a dependency map: read it bottom-up. Each box is one symbol you just learned, and an arrow "X → Y" means X is needed to build Y. Start at the clock $t$ and the action $A_t$ (top-left inputs); these produce rewards; the counter $k$, powers, and $\gamma$ weight those rewards; the sigma $\sum$ adds them into the return $G_t$; the geometric series bounds it; the recursion re-folds it — and all of that together is the topic.

Concretely: the box "Return G at t" is section 7, "Bounded return" is section 8, and "Recursion" is section 9. Follow any arrow into the bottom box "Discount factor and returns" and you are reading the parent note Discount factor and returns (5.1.6). All of this lives inside a Markov Decision Process, the formal world where states, actions and rewards are defined.

Equipment checklist

Test yourself — cover the right side and answer each aloud before revealing.