Exercises — Agent, environment, state, action, reward

return $G_t$ and the discount factor $\gamma$? The return $G_t$ is the total future reward the agent collects starting at time $t$: $G_t=r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+\cdots$. The discount factor $\gamma \in [0,1)$ is a number that shrinks rewards the further they are in the future — think of it as "how much a reward one step from now is worth compared to right now." $\gamma =0$ means "I only care about this instant"; $\gamma \to 1$ means "I weigh the far future almost as much as now."

(a) action — the only thing the agent directly picks. (b) environment — everything outside the agent that reacts. (c) state — the complete description of the situation at that moment. (d) agent — the learner/decision-maker (the "brain"). (e) reward — the scalar feedback signal encoding the goal.

In state $s_t$ the agent first chooses action $a_t$ (from its policy). The environment then does two things: it emits reward $r_t=R(s_t,a_t,s_{t+1})$ and transitions to $s_{t+1}\sim P(s^{\prime }\mid s_t,a_t)$. In the written trajectory $r_t$ appears before $s_{t+1}$, but both are produced by the environment in response to the same action $a_t$. The agent controls only $a_t$; state and reward "emerge" from it.

Finite-horizon return is a plain sum from time $t$ onward. $G_0=r_0+r_1+r_2+r_3=2+0+(-1)+5=6.$ $G_2=r_2+r_3=-1+5=4.$ Notice $G_0\ne G_2$: the return depends on where you start counting.

Each future reward is multiplied by ${\gamma }^k$ where $k$ is how many steps ahead it is. $G_0=2+0.5(0)+0.5^2(-1)+0.5^3(5)$ $=2+0-0.25+0.125\cdot 5=2-0.25+0.625=\mathrm{2.375.}$

This is a constant reward, so $G_0=3{\sum }_{k=0}^{\infty }{\gamma }^k$. The geometric series sums to $\frac{1}{1-\gamma }$ for $|\gamma |<1$. $G_0=3\cdot \frac{1}{1-0.9}=3\cdot \frac{1}{0.1}=3\cdot 10=30.$ Even an infinite stream of rewards gives a finite return — that is exactly why discounting is used. See Bellman Equations for where this geometric collapse reappears.

Apply $G_0={\sum }_{k=0}^3{\gamma }^k{r}_k$ with $\gamma =0.9$: $G_0=-1+0.9(-1)+0.9^2(-5)+0.9^3(10).$ Compute each term: $0.9^2=0.81$, $0.9^3=0.729$. $=-1-0.9-4.05+7.29=\mathrm{1.34.}$ The wall bump at $k=2$ costs the most ($-4.05$ after discount); the goal reward, though large, is shrunk to $7.29$ by being 3 steps away. Net positive, so this path was still worthwhile.

From the parent's convergence argument, $|G_t|\le \frac{R_{\max }}{1-\gamma }$. $|G_t|\le \frac{10}{1-0.99}=\frac{10}{0.01}=1000.$ Interpretation: with $\gamma$ this close to 1 the agent is extremely far-sighted — its return can be up to 100× the single-step maximum. Learning is harder here because a value estimate must sum the effect of ~100 future steps.

With $0$ per step, the only nonzero term in $G_0$ is the discounted goal reward ${\gamma }^T\cdot 10$, where $T$ is the number of steps taken. If $\gamma =1$ (no discount), then ${\gamma }^T=1$ for any $T$ — reaching the goal in 5 steps or 500 steps gives the identical return $10$. So the agent is indifferent to speed and may wander forever. Subtlety: if $\gamma <1$, then ${\gamma }^T$ shrinks with larger $T$, so discounting alone gives a mild "hurry up" pressure. The explicit $-1$ step penalty in the parent is a stronger, $\gamma$-independent way to force efficiency. This is a design choice about shaping the goal — see Policy and Value Functions.

Peel off the $k=0$ term: $G_t={\gamma }^0{r}_t+{\sum }_{k=1}^{\infty }{\gamma }^k{r}_{t+k}=r_t+{\sum }_{k=1}^{\infty }{\gamma }^k{r}_{t+k}.$ Factor $\gamma$ out of the remaining sum and re-index with $j=k-1$: ${\sum }_{k=1}^{\infty }{\gamma }^k{r}_{t+k}=\gamma {\sum }_{k=1}^{\infty }{\gamma }^{k-1}{r}_{t+k}=\gamma {\sum }_{j=0}^{\infty }{\gamma }^j{r}_{(t+1)+j}=\gamma G_{t+1}.$ Therefore $G_t=r_t+\gamma G_{t+1}$. This one line is the seed of the Bellman Equations.

(a) Direct. Split into the two special terms plus the constant tail: $G_0=2+0.5(2)+\underset{\text{tail}}{\underbrace{\sum _{k=2}^{\infty }0.5^k\cdot 1}}.$ First two: $2+1=3$. The tail is a geometric series starting at $k=2$: ${\sum }_{k=2}^{\infty }0.5^k=0.5^2\cdot \frac{1}{1-0.5}=0.25\cdot 2=\mathrm{0.5.}$ So $G_0=3+0.5=3.5$.

(b) Recursion check. First find $G_1$. From step 1 onward the rewards are $2,1,1,1,\dots$ $G_1=2+{\sum }_{k=1}^{\infty }0.5^k\cdot 1=2+0.5\cdot \frac{1}{1-0.5}=2+1=3.$ Then $G_0=r_0+\gamma G_1=2+0.5(3)=3.5$. ✓ Same answer.

No. The true environment state includes the opponents' hidden cards and the deck order; the agent's observation (its two cards) is not a complete description. The future — who wins the pot — depends on information the agent cannot see, so knowing only the observation does not make the future conditionally independent of the past. This is a partially observable problem, a Partial Observability (POMDP), not a plain Markov Decision Process (MDP). The fix in practice is to fold history (past actions/bets) into an augmented "agent state" to recover an approximately-Markov signal.

With a single step, the return equals the single reward: $G_0=r_0$. Its expectation is the payout probability. $\mathbb{E}[G_0\mid \text{arm A}]=0.6,\mathbb{E}[G_0\mid \text{arm B}]=\mathrm{0.4.}$ Optimal policy: always pull arm A (expected $0.6>0.4$). Why $\gamma$ doesn't matter: with $T=0$ there is exactly one reward and no future terms, so no reward ever gets multiplied by ${\gamma }^k$ for $k\ge 1$. Discounting only bites when there is a future. The bandit's real difficulty is instead not knowing the $0.6$/$0.4$ in advance — that's Exploration vs Exploitation.

Plug $\gamma =0$ into $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}$. Every term with $k\ge 1$ has a factor $0^k=0$; the $k=0$ term has $0^0=1$. So $G_t=r_t.$ Maximizing return now means maximizing only the immediate reward $r_t$ — a greedy / myopic agent that ignores all consequences. This is the opposite extreme from Exercise 3.2's far-sighted $\gamma =0.99$ agent. Real tasks pick $\gamma$ between these poles to trade off short vs long-term payoff.

(a) Deviation term: $-(19.5-21{)}^2=-(-1.5{)}^2=-2.25$. Energy term: $-0.01(0.8{)}^2=-0.01\cdot 0.64=-0.0064$. $r_t=-2.25-0.0064=-2.2564\approx -\mathrm{2.256.}$ (b) All rewards are $\le 0$, so "maximizing" means pushing the reward toward zero — i.e. getting the temperature as close to $21^{\circ }$ as possible while using as little heater power as possible. The best achievable per-step reward is $0$ (perfect temperature, zero power), which is never quite reached, so the agent keeps improving. (c) The action $a_t\in [0,1]$ is continuous. A continuous action space has infinitely many actions, so you cannot store one value per (state, action) pair in a table — tabular/lookup methods break. You need function approximation (see Model-Free vs Model-Based RL) or a policy that outputs a real number directly.

Ordering in a trajectory ::: $s_t\to a_t\to (r_t,s_{t+1})$; only $a_t$ is the agent's. Return exponent rule ::: ${\gamma }^k$ where $k$ = steps ahead of the start time $t$. Convergence bound ::: $|G_t|\le R_{\max }/(1-\gamma )$. Recursion ::: $G_t=r_t+\gamma G_{t+1}$. $\gamma =0$ vs $\gamma \to 1$ ::: myopic (immediate reward) vs far-sighted. Observation $\ne$ state when ::: information is hidden (POMDP).