Discount factor and returns

WHAT are we defining?

• $\gamma = 0$ → myopic agent: only the immediate reward $R_{t+1}$ matters.
• $\gamma \to 1$ → far-sighted agent: distant rewards count almost as much as near ones.

WHY do we discount at all?

Three independent reasons, all pointing the same way:

1. Mathematical convergence. In tasks that never end (continuing tasks), $\sum R$ could be infinite. Discounting with $\gamma < 1$ keeps the sum finite (proved below).
2. Uncertainty about the future. Think of $\gamma$ as a "survival probability" — at each step there's a chance the process stops. Reward you might never live to collect should count less.
3. Preference / economics. Like interest rates: money (reward) now can be reinvested, so it is genuinely worth more than the same amount later.

HOW: derive the bounded-return result from scratch

Claim: If every reward is bounded, $|R| \le R_{\max}$, and $0 \le \gamma < 1$, then $G_t$ is finite.

Step 1 — Bound each term. $|G_t| = \left|\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}\right| \le \sum_{k=0}^{\infty}\gamma^k |R_{t+k+1}| \le \sum_{k=0}^{\infty}\gamma^k R_{\max}$ Why this step? Triangle inequality moves the absolute value inside, then we replace every reward by its worst case $R_{\max}$.

Step 2 — Sum the geometric series. Let $S = \sum_{k=0}^{\infty}\gamma^k = 1 + \gamma + \gamma^2 + \cdots$. Multiply by $\gamma$: $\gamma S = \gamma + \gamma^2 + \cdots \implies S - \gamma S = 1 \implies S = \frac{1}{1-\gamma}$ Why this step? Subtracting a shifted copy telescopes all the middle terms — the classic geometric-series trick, valid because $\gamma<1$ makes $\gamma^k\to0$.

Step 3 — Combine. $\boxed{\;|G_t| \le \frac{R_{\max}}{1-\gamma}\;}$ So the return is finite. Notice: as $\gamma \to 1$, this bound blows up — long horizons need care.

Effective horizon (the 80/20 of $\gamma$)

The weights $\gamma^k$ shrink geometrically. Two related "horizon" numbers show up — don't confuse them:

1. The $\tfrac{1}{1-\gamma}$ scale. This is the value of $\sum_{k=0}^\infty \gamma^k$ (total weight) and the characteristic scale over which weights decay to $1/e$-ish. It's the number usually quoted as the rough planning horizon.

2. The true mean step you look at. Normalize the weights into a probability distribution $P(k)=(1-\gamma)\gamma^k$ (they sum to 1). Its mean index is $\mathbb{E}[k] = \sum_{k=0}^{\infty} k\,(1-\gamma)\gamma^k = \frac{\gamma}{1-\gamma}.$ Why this step? Using $\sum_k k\,\gamma^k = \dfrac{\gamma}{(1-\gamma)^2}$ and multiplying by the normalizer $(1-\gamma)$ gives $\dfrac{\gamma}{1-\gamma}$ — note the extra $\gamma$ in the numerator, so the mean is one less than $\tfrac{1}{1-\gamma}$.

$\boxed{\;H_{\text{scale}}=\frac{1}{1-\gamma}\quad\text{(weight scale)},\qquad \mathbb{E}[k]=\frac{\gamma}{1-\gamma}\quad\text{(mean step)}\;}$

Worked examples

Common mistakes (Steel-manned)

Flashcards

Connections

• Markov Decision Process — returns are defined over MDP trajectories.
• Value functions and Bellman equations — $v(s)=\mathbb{E}[G_t\mid S_t=s]$ builds directly on $G_t = R_{t+1}+\gamma G_{t+1}$.
• Episodic vs continuing tasks — decides whether $\gamma=1$ is allowed.
• Reward hypothesis — what the $R_t$'s represent.
• Geometric series — the math powering convergence and the horizon.
• Temporal Difference learning — uses the recursive return to bootstrap.

Concept Map

Hinglish (regional understanding)