Level 1 — RecognitionDeep & Advanced RL

Deep & Advanced RL

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Chapter: 5.2 Deep & Advanced Reinforcement Learning Level: 1 — Recognition (MCQ, Matching, True/False with justification) Time limit: 20 minutes Total marks: 30

Section A — Multiple Choice (1 mark each, 10 marks)

Q1. In a Deep Q-Network, the neural network is used to approximate:

(a) the state transition probabilities $P(s'|s,a)$
(b) the action-value function $Q(s,a)$
(c) the reward function $R(s,a)$
(d) the policy entropy $H(\pi)$

Q2. The primary purpose of experience replay in DQN is to:

(a) increase the discount factor over time
(b) break temporal correlations between consecutive samples
(c) guarantee an optimal policy in one update
(d) remove the need for a target network

Q3. The target network in DQN is updated:

(a) every gradient step, identically to the online network
(b) periodically (or via slow soft updates), lagging the online network
(c) only at the start of training
(d) using a completely different reward signal

Q4. Double DQN primarily addresses which problem of standard DQN?

(a) exploding gradients
(b) overestimation bias of Q-values
(c) sparse rewards
(d) high memory usage of the replay buffer

Q5. In the Dueling DQN architecture, the network splits into two streams that estimate:

(a) policy and value
(b) state value $V(s)$ and advantage $A(s,a)$
(c) reward and transition
(d) mean and variance of returns

Q6. The REINFORCE algorithm updates policy parameters using:

(a) the TD error from a critic
(b) the Monte Carlo return along a sampled trajectory
(c) a clipped surrogate objective
(d) a KL-divergence trust region constraint

Q7. In an actor-critic method, the critic is responsible for:

(a) selecting actions to execute
(b) estimating a value function to evaluate the actor
(c) storing past transitions
(d) copying weights to the target network

Q8. PPO keeps policy updates stable primarily by:

(a) a hard KL constraint solved with conjugate gradient
(b) a clipped probability-ratio surrogate objective
(c) removing the advantage function
(d) using experience replay of off-policy data

Q9. Soft Actor-Critic (SAC) augments the RL objective with:

(a) an entropy term encouraging exploration
(b) a reward-shaping potential
(c) a target-policy smoothing noise only
(d) a discrete action softmax constraint

Q10. Potential-based reward shaping with $F(s,s') = \gamma\Phi(s') - \Phi(s)$ is valuable because it:

(a) always speeds up learning regardless of $\Phi$
(b) preserves the optimal policy of the original MDP
(c) removes the discount factor
(d) converts the problem to a bandit

Section B — Matching (1 mark each, 8 marks)

Q11. Match each method (i–viii) to its defining characteristic (A–H).

#	Method		Characteristic
i	TRPO	A	On-policy method with clipped surrogate objective
ii	A3C	B	Enforces a KL-divergence trust region via constrained optimization
iii	PPO	C	Off-policy, maximum-entropy actor-critic
iv	SAC	D	Asynchronous parallel actors updating a shared network
v	Experience replay	E	Learns dynamics model to plan/generate samples
vi	Model-based RL	F	Stores transitions for reuse and decorrelation
vii	Target network	G	Multiple learning agents interacting in shared environment
viii	Multi-agent RL	H	Slowly-updated copy providing stable TD targets

Write pairs, e.g. i–B.

Section C — True/False WITH Justification (2 marks each, 12 marks)

1 mark correct T/F, 1 mark valid justification.

Q12. DQN can be applied directly to continuous action spaces without modification.

Q13. The A2C algorithm uses the advantage $A(s,a)=Q(s,a)-V(s)$ to reduce the variance of the policy-gradient estimate.

Q14. Sparse-reward environments are generally harder for RL because the agent rarely receives an informative learning signal.

Q15. Increasing the target-network update frequency to every step makes DQN more stable.

Q16. Policy gradient methods can naturally learn stochastic policies, whereas value-based methods like DQN produce deterministic greedy policies.

Q17. Model-based RL is always more sample-efficient AND more asymptotically accurate than model-free RL.

Answer keyMark scheme & solutions

Section A (1 mark each)

Q1 — (b). A DQN parameterizes $Q(s,a;\theta)$ with a neural net; it is value-based, not a model of dynamics/reward.

Q2 — (b). Sampling random minibatches from the buffer breaks the strong correlation between consecutive online transitions, stabilizing SGD.

Q3 — (b). The target net is a periodically-copied (or Polyak-averaged) lagging network providing stationary bootstrap targets.

Q4 — (b). Double DQN decouples action selection (online net) from action evaluation (target net), reducing max-operator overestimation bias.

Q5 — (b). Dueling splits into value stream $V(s)$ and advantage stream $A(s,a)$ , recombined as $Q=V+(A-\overline{A})$ .

Q6 — (b). REINFORCE is Monte Carlo: $\nabla_\theta J=\mathbb{E}[G_t\nabla_\theta\log\pi_\theta(a_t|s_t)]$ .

Q7 — (b). The critic estimates $V$ or $Q$ to evaluate/bootstrap the actor's actions.

Q8 — (b). PPO uses the clipped surrogate $\min(r_t\hat A_t,\ \text{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t)$ .

Q9 — (a). SAC maximizes reward plus entropy $\mathbb{E}[\sum \gamma^t(r_t+\alpha H(\pi(\cdot|s_t)))]$ .

Q10 — (b). Potential-based shaping is policy-invariant: it preserves the optimal policy of the original MDP (Ng et al.).

Section B (1 mark each)

Q11:

i–B (TRPO → KL trust region)
ii–D (A3C → asynchronous parallel actors)
iii–A (PPO → clipped surrogate)
iv–C (SAC → off-policy max-entropy actor-critic)
v–F (Experience replay → stores transitions)
vi–E (Model-based → learns dynamics)
vii–H (Target network → slow copy for TD targets)
viii–G (Multi-agent → multiple agents)

Section C (2 marks each: 1 T/F + 1 justification)

Q12 — FALSE. DQN requires $\max_a Q$ over actions, tractable only for discrete/finite action sets; continuous actions need methods like DDPG/SAC.

Q13 — TRUE. Subtracting the baseline $V(s)$ (advantage) leaves the gradient unbiased while reducing its variance, the core of A2C.

Q14 — TRUE. With sparse rewards the agent gets almost no gradient signal from most transitions, making credit assignment and exploration difficult.

Q15 — FALSE. Updating the target every step makes targets move with the online net (like no target net), reintroducing instability/divergence; the lag is what stabilizes.

Q16 — TRUE. Policy gradients output a distribution $\pi_\theta(a|s)$ (can be stochastic); DQN's $\arg\max_a Q$ is deterministic (barring $\epsilon$ -greedy exploration).

Q17 — FALSE. Model-based RL is often more sample-efficient, but model bias can cap asymptotic performance below strong model-free methods; "always... AND more accurate" is incorrect.

[
  {"claim": "Dueling recombination Q = V + (A - mean(A)) preserves Q when A already centered",
   "code": "V=Symbol('V'); A1,A2=symbols('A1 A2'); Amean=(A1+A2)/2; Q1=V+(A1-Amean); Q2=V+(A2-Amean); result = simplify((Q1-Q2)-(A1-A2))==0"},
  {"claim": "Advantage baseline subtraction leaves policy gradient unbiased: E[b * grad log pi] = 0 for sum pi = 1",
   "code": "p1,p2=symbols('p1 p2',positive=True); b=Symbol('b'); dp1,dp2=symbols('dp1 dp2'); expr = b*(dp1+dp2); result = simplify(expr.subs({dp1: -dp2}))==0"},
  {"claim": "Potential shaping term F = gamma*Phi_next - Phi is zero-sum telescoping over a return so optimal policy invariant (single-step check gamma=1 returns to same state)",
   "code": "g=Symbol('gamma'); Ph=Symbol('Phi'); F=g*Ph-Ph; result = simplify(F.subs(g,1))==0"}
]