Level 1 — RecognitionReinforcement Learning Foundations

Reinforcement Learning Foundations

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Difficulty Level: 1 — Recognition (MCQ, Matching, True/False with justification) Time Limit: 20 minutes Total Marks: 30

Section A — Multiple Choice (1 mark each, 10 marks)

Choose the single best answer.

Q1. In the RL loop, what does the environment return to the agent after an action?

(a) Only a reward
(b) A next state and a reward
(c) A policy
(d) A value function

Q2. The Markov property states that the next state depends on:

(a) The entire history of states and actions
(b) Only the current state and action
(c) Only the reward
(d) The discount factor

Q3. A policy $\pi(a\mid s)$ is best described as:

(a) A mapping from states to expected returns
(b) A mapping from states to actions (or a distribution over actions)
(c) The immediate reward for an action
(d) The transition probability function

Q4. The state-value function $V^\pi(s)$ represents:

(a) The immediate reward at state $s$
(b) The expected return starting from $s$ and following $\pi$
(c) The best possible action at $s$
(d) The probability of reaching $s$

Q5. The discount factor $\gamma$ typically satisfies:

(a) $\gamma > 1$
(b) $\gamma < 0$
(c) $0 \le \gamma \le 1$
(d) $\gamma = 2$

Q6. Which update rule is off-policy?

(a) SARSA
(b) Q-learning
(c) Monte Carlo on-policy control
(d) Policy evaluation

Q7. In $\epsilon$ -greedy with $\epsilon = 0.1$ and 4 possible actions, the probability of choosing the current greedy action is:

(a) $0.10$
(b) $0.90$
(c) $0.925$
(d) $1.00$

Q8. Monte Carlo methods update value estimates:

(a) After every single time step
(b) Only after a complete episode terminates
(c) Using the transition model $P$
(d) Without any returns

Q9. The TD(0) target for updating $V(s_t)$ is:

(a) $r_{t+1}$
(b) $G_t$ (full return)
(c) $r_{t+1} + \gamma V(s_{t+1})$
(d) $V(s_t)$

Q10. Policy iteration alternates between:

(a) Exploration and exploitation
(b) Policy evaluation and policy improvement
(c) Sampling and bootstrapping
(d) Reward and penalty

Section B — Matching (1 mark each, 8 marks)

Q11–Q18. Match each term in Column X with the correct description in Column Y. Write the pairs (e.g., 11 → C).

Column X	Column Y
Q11. Agent	A. Long-term accumulated (discounted) reward
Q12. Return $G_t$	B. Expected return for a state–action pair
Q13. Bellman equation	C. The learner/decision-maker
Q14. Action-value $Q^\pi(s,a)$	D. Recursive relation for value functions
Q15. Exploration	E. Choosing the currently best-known action
Q16. Exploitation	F. Trying new actions to gather information
Q17. SARSA	G. On-policy TD control algorithm
Q18. Value iteration	H. DP using Bellman optimality with one sweep of improvement

Section C — True/False with Justification (2 marks each: 1 for T/F, 1 for reason, 12 marks)

Q19. A larger discount factor $\gamma$ makes the agent more short-sighted (values immediate rewards more). (True/False + justify)

Q20. Q-learning learns the value of the policy it is currently following. (True/False + justify)

Q21. Dynamic programming methods require a known model of the environment (transition probabilities and rewards). (True/False + justify)

Q22. In an MDP, the reward can depend on the state, action, and next state. (True/False + justify)

Q23. Temporal Difference learning cannot begin updating until an episode finishes. (True/False + justify)

Q24. An $\epsilon$ -greedy policy with $\epsilon = 0$ never explores. (True/False + justify)

Answer keyMark scheme & solutions

Section A — MCQ (1 mark each)

Q	Ans	Why
Q1	(b)	The environment emits the next state $s_{t+1}$ and scalar reward $r_{t+1}$ in response to $a_t$ .
Q2	(b)	Markov property: future depends only on current state (and action), not full history.
Q3	(b)	A policy maps states to actions or a probability distribution over actions.
Q4	(b)	$V^\pi(s)=\mathbb{E}_\pi[G_t\mid s_t=s]$ — expected return under $\pi$ .
Q5	(c)	Discount factor is bounded in $[0,1]$ to keep returns finite and well-defined.
Q6	(b)	Q-learning uses $\max_a Q(s',a)$ (greedy/target policy) while behaving differently → off-policy.
Q7	(c)	Greedy prob $= (1-\epsilon) + \epsilon/
Q8	(b)	MC uses complete-episode returns $G_t$ , so it needs episode termination.
Q9	(c)	TD(0) target bootstraps: $r_{t+1}+\gamma V(s_{t+1})$ .
Q10	(b)	Policy iteration = evaluate current policy, then improve greedily; repeat.

Section A total: 10 marks

Section B — Matching (1 mark each)

Q	Match	Why
Q11	C	Agent = the decision-maker/learner.
Q12	A	Return = discounted sum of future rewards.
Q13	D	Bellman equation = recursive relation for value functions.
Q14	B	$Q^\pi(s,a)$ = expected return for a state–action pair.
Q15	F	Exploration = trying new actions for information.
Q16	E	Exploitation = choosing the best-known action.
Q17	G	SARSA = on-policy TD control.
Q18	H	Value iteration uses Bellman optimality with combined evaluation/improvement sweep.

Section B total: 8 marks

Section C — True/False with Justification (2 marks each)

Q19. FALSE. (1) A larger $\gamma$ (closer to 1) makes the agent more far-sighted, weighting future rewards heavily; a small $\gamma$ makes it short-sighted. (1 for reason)

Q20. FALSE. (1) Q-learning is off-policy: it learns the optimal action-value function using $\max_a Q(s',a)$ regardless of the behavior (e.g., $\epsilon$ -greedy) policy actually followed. (1)

Q21. TRUE. (1) DP (value/policy iteration) needs the full model — transition probabilities $P(s'\mid s,a)$ and reward function — to compute Bellman backups. (1)

Q22. TRUE. (1) The general reward form is $r(s,a,s')$ ; rewards may depend on current state, action, and resulting next state. (1)

Q23. FALSE. (1) TD learning updates online, at each step, using bootstrapped estimate $r_{t+1}+\gamma V(s_{t+1})$ ; it does not wait for the episode to end (that is Monte Carlo). (1)

Q24. TRUE. (1) With $\epsilon=0$ , the policy is fully greedy — it always exploits and never selects a random exploratory action. (1)

Section C total: 12 marks Grand total: 30 marks

[
  {"claim":"Q7: greedy action probability = 0.925 for eps=0.1, 4 actions","code":"eps=Rational(1,10); A=4; p=(1-eps)+eps/A; result=(p==Rational(37,40)) and (float(p)==0.925)"},
  {"claim":"Q7: each non-greedy action prob = eps/A = 0.025, and total sums to 1","code":"eps=Rational(1,10); A=4; greedy=(1-eps)+eps/A; other=eps/A; total=greedy+(A-1)*other; result=(other==Rational(1,40)) and (total==1)"},
  {"claim":"Discount bound: for gamma in [0,1), sum of gamma**t of constant reward 1 converges to 1/(1-gamma)","code":"g=symbols('g'); t=symbols('t'); S=summation(g**t,(t,0,oo)); val=S.rewrite(Piecewise); result=(simplify(S.subs(g,Rational(1,2)))==2)"},
  {"claim":"TD target with r=1, gamma=0.9, V(s')=10 equals 10","code":"r=1; gamma=Rational(9,10); Vnext=10; target=r+gamma*Vnext; result=(target==10)"}
]