Question bank — Discount factor and returns

True. The action chosen at time $t$ produces its reward one tick later, so the earliest reward "caused by" being at $t$ is $R_{t+1}$; starting at $R_t$ is the classic off-by-one that corrupts every Bellman equation.

False. $\gamma$ amplifies the future, whatever its sign. If future rewards are penalties, a bigger $\gamma$ makes $G_t$ smaller, not larger.

False. ${\gamma }^k=0$ for every $k\ge 1$, so only $R_{t+1}$ survives — the agent is fully myopic and the entire future beyond one step is invisible.

False. It is only safe for episodic tasks guaranteed to terminate; in continuing tasks the bound $R_{\max }/(1-\gamma )$ blows up and $G_t$ can diverge.

False (when $\gamma <1$). The geometric series sums to $\frac{1}{1-\gamma }$, a finite number (e.g. $10$ at $\gamma =0.9$). Only $\gamma =1$ makes it diverge.

False. $\frac{1}{1-\gamma }$ is the total weight / decay scale; the true mean step index under normalized weights is $\frac{\gamma }{1-\gamma }$ — exactly one less.

False. The received reward is unchanged; ${\gamma }^k$ is a preference weighting inside the objective, expressing that we value later reward less when deciding.

False on its own. You also need $\gamma <1$. With $|R|\le R_{\max }$ and $\gamma =1$ the sum still diverges — both conditions are required for $|G_t|\le \frac{R_{\max }}{1-\gamma }$.

False. It is the same object; you just peel off the first term and factor $\gamma$ out of the rest. They are algebraically identical.

The indices are shifted down by one. The correct first reward is $R_{t+1}$, giving $G_t=R_{t+1}+\gamma R_{t+2}+\cdots$.

Only for $|\gamma |<1$. At $\gamma =1$ the partial sums grow without bound; the closed form is meaningless there.

The opposite: as $\gamma \to 1$ the denominator $\to 0$, so the bound explodes. Long horizons need care, not comfort.

You must work backward from a terminal condition. Without $G_{\text{terminal}}=0$ the recursion has no base case and can't be evaluated.

They don't sum to 1; you must multiply by $(1-\gamma )$. The correct distribution is $P(k)=(1-\gamma ){\gamma }^k$.

Backwards. Survival at each step has probability $\gamma$; the stopping probability is $1-\gamma$. Higher $\gamma$ means more likely to keep going.

Decay requires $\gamma <1$ strictly. At $\gamma =1$, ${\gamma }^k=1$ for all $k$ — no decay at all — so convergence fails.

Because the reward is a consequence of the action taken at $t$, arriving on the next tick; this convention keeps $G_t=R_{t+1}+\gamma G_{t+1}$ clean and consistent across all Bellman equations.

Three converging reasons: it keeps sums finite in continuing tasks, it models uncertainty about reaching the future, and it encodes a genuine preference for sooner reward (like interest). See the Reward hypothesis for what "reward" is meant to capture.

Subtracting the shifted copy $\gamma S$ telescopes the middle terms only if ${\gamma }^k\to 0$; that vanishing tail is exactly what $\gamma <1$ guarantees.

It is self-referential — the return today defined in terms of the return tomorrow — and this is the exact seed of all value-function recursions and of Temporal Difference learning.

They differ by exactly $1$, and as $\gamma \to 1$ both grow huge so the gap becomes negligible relative to their size — the loose "$\frac{1}{1-\gamma }$ horizon" survives on that approximation.

Turning it toward $0$ makes the agent grab immediate reward (distrusting the future); toward $1$ makes it patiently plan long-term (trusting the future), continuously trading between the two.

A recursion needs a base case to bottom out; declaring "nothing comes after the last reward" ($G=0$) lets the backward computation start, mirroring how dynamic programming seeds its values.

$G_t=R_{t+1}$ exactly — purely myopic; every future term is annihilated, so a huge later payoff is completely ignored.

Yes. With guaranteed termination the sum has finitely many nonzero terms, so no discounting is needed for convergence — this is the one setting where $\gamma =1$ is legitimate.

The return can diverge to $\pm \infty$; the standard fix is $\gamma <1$ or an average-reward formulation instead.

It contributes $0.9^{49}\cdot 100\approx 0.6$ — heavily discounted to under one point, so a distant windfall barely moves $G_t$ at $\gamma =0.9$.

Yes. Bounded rewards with $\gamma <1$ always converge; here $|G_t|\le \frac{1}{1-\gamma }$, and the alternation just makes the sum settle to $\frac{1}{1+\gamma }$.

$G_t=R_{t+1}=0$ and $G_{t+1}=0$; the whole return collapses to zero — a valid degenerate episode, not an error.

Both $\frac{1}{1-\gamma }$ and $\frac{\gamma }{1-\gamma }$ grow without bound, so the agent's planning window stretches toward the whole infinite future — the far-sighted limit central to the Markov Decision Process setup.