Question bank — Monte Carlo methods

False — MC is model-free; it averages actual observed returns and never touches $P(s^{\prime }\mid s,a)$ or $R(s,a)$.

False — correlation inflates variance, not bias; every-visit is a mean of returns each with expectation $v_{\pi }(s)$, so it is unbiased at every finite sample size (first-visit is unbiased too, with cleaner independence).

False — MC requires complete episodes that terminate, because the return $G_t=R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{T-t-1}{R}_T$ can't be computed until an episode ends; continuing tasks have no $T$.

False — MC uses the whole actual return $G_t$, never the estimate $V(S_{t+1})$. Updating from another estimate is bootstrapping (that's TD, not MC).

True — $G_t=-1-1-\cdots -1=-(T-t)$, so the return literally counts remaining steps, which is why it measures "distance to goal".

False — they agree only when returns from repeat visits happen to match; in general every-visit folds in extra (correlated) samples from one episode and a different average.

True — as figure s02 shows, $\frac{1}{n}{\sum }_i{G}_i=\frac{1}{n}\left[(n-1)\text{oldmean}+G_n\right]=\text{oldmean}+\frac{1}{n}(G_n-\text{oldmean})$, so it is memory-efficient and exactly equal to recomputing the full average.

In general no — it always keeps probability $\epsilon$ on random actions, so it's the best policy within the $\epsilon$-soft class, not the true greedy optimum unless every non-greedy action happens to be tied.

True — greedy improvement $\arg {\max }_{a}Q(s,a)$ needs no model, whereas improving from $V$ requires $P$ and $R$ to look one step ahead, which model-free MC doesn't have.

False — a larger $\gamma$ weights more distant, more stochastic future rewards, which typically increases return variance; discounting closer to 0 shrinks it.

That's a TD(0) update, not MC. MC uses the full return $G_t$ and updates only after the episode ends, never mid-episode from a successor's estimate.

Wrong scope — it's per-state: for each state $s$, use only $s$'s first occurrence in that episode, not just the episode's very first state.

No — with $\epsilon =0$ some $(s,a)$ pairs may never be visited, so their $Q$ never updates; convergence proofs need every action tried infinitely often (exploration).

Unbiased ≠ fast. MC has high variance, so it often converges slower in practice; bias and variance trade off, and TD's low variance frequently wins.

Backwards — $1/N(s)$ gives equal weight to all past returns (a true mean); a constant $\alpha$ is what weights recent returns more, tracking non-stationary values.

No — $G_t=R_{t+1}+\gamma R_{t+2}+\cdots$ starts at $R_{t+1}$; it's the reward from time $t$ onwards, excluding what already happened before arriving.

Reversed — repeat visits within one episode are correlated, which is why first-visit (one independent sample per episode) is the theoretically cleaner one.

Because the return $G_t$ is a sum running to the terminal step $T$; without termination there's no final value to sum to, so no complete sample return exists to average.

Each MC return is a single sampled path through many stochastic transitions and rewards, so noise compounds along the whole trajectory; TD only samples one step then trusts an estimate.

It lets us compute returns backwards through an episode in one pass — a pure arithmetic convenience — using the already-computed $G_{t+1}$, not a learned estimate.

$Q(s,a)$ already bakes in the expected discounted future of taking $a$ then following $\pi$, so choosing the best $Q$ needs no $P$ or $R$ — the model information is implicit in the samples.

By the law of large numbers the sample mean of returns approaches the expected return, and ${\mathbb{E}}_{\pi }[G_t\mid S_t=s]$ is the definition of $v_{\pi }(s)$.

The weights ${\gamma }^k$ form a geometric series; bounded rewards times $\sum {\gamma }^k=1/(1-\gamma )$ give a finite bound regardless of how many steps occur.

Evaluation just measures a fixed policy $\pi$, following it is enough; control must discover better actions, so it needs to sometimes try non-greedy actions to gather $Q$ data on them.

It stays at its initial value with $N(s)=0$; MC provides no information about states experience never reaches, a real coverage limitation.

$G_t=R_{t+1}$ only — MC then estimates just the expected immediate reward (a myopic value), ignoring everything downstream.

$G_0=R_1$, so $V(S_0)$ moves toward $R_1$; first- and every-visit coincide trivially because $S_0$ appears once.

First-visit adds one sample $G_a$ (earliest occurrence); every-visit adds two samples $G_a$ and $G_b$ from the single episode.

The return from a terminal state is $0$ (no future rewards) and its value is fixed at $0$ by definition — we don't estimate it.

One — with no randomness every episode gives the identical exact return, so a single sample equals the true value (variance is zero).

That $(s,a)$ pair is never sampled, so its $Q$ never updates — exactly why on-policy control uses $\epsilon$-soft policies that keep every action's probability strictly positive.

Not to a fixed point exactly — a constant $\alpha$ keeps chasing new returns and leaves residual variance; it's used deliberately for non-stationary problems where old data should fade.