Level 2 — RecallDeep & Advanced RL

Deep & Advanced RL

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Level: 2 (Recall: definitions, standard textbook problems, short derivations) Time Limit: 30 minutes Total Marks: 40

Q1. State the DQN loss function used to train the Q-network, defining every symbol. Explain briefly why a target network with parameters $\theta^-$ is used instead of the online parameters $\theta$ . (4 marks)

Q2. Explain the purpose of experience replay in DQN. List two concrete benefits it provides during training. (3 marks)

Q3. In standard DQN the TD target is $r + \gamma \max_a Q(s',a;\theta^-)$ . Write the Double DQN target and state, in one sentence, what problem it fixes. (4 marks)

Q4. Describe the Dueling DQN architecture. Write the aggregation equation that combines the value stream $V(s)$ and advantage stream $A(s,a)$ , and state why the mean-subtraction term is required. (4 marks)

Q5. State the REINFORCE policy-gradient update rule. Starting from the policy gradient theorem $\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\, Q^\pi(s,a)\big],$ explain in one line how REINFORCE estimates $Q^\pi(s,a)$ . (4 marks)

Q6. Define the advantage function $A^\pi(s,a)$ in terms of $Q^\pi$ and $V^\pi$ . Explain in two sentences how the actor–critic method uses the critic to reduce variance compared to REINFORCE. (4 marks)

Q7. Write the PPO clipped surrogate objective, defining the probability ratio $r_t(\theta)$ and the clipping parameter $\epsilon$ . State why clipping is preferred over TRPO's hard KL constraint. (5 marks)

Q8. Soft Actor-Critic (SAC) optimizes a maximum-entropy objective. Write the entropy-regularized objective it maximizes and state, in one sentence, the practical benefit of the entropy term. (4 marks)

Q9. Numerical / short computation: Given a single episode with rewards $r_1=1,\ r_2=0,\ r_3=2$ and discount factor $\gamma=0.9$ , compute the discounted return $G_1$ (return from the first step) used by REINFORCE. Show working. (4 marks)

Q10. Distinguish between model-based and model-free RL in one sentence each, and give one example algorithm of each type. Briefly explain what reward shaping does for sparse-reward problems. (4 marks)

End of paper.

Answer keyMark scheme & solutions

Q1. (4 marks) Loss: $L(\theta) = \mathbb{E}_{(s,a,r,s')\sim D}\Big[\big(r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta)\big)^2\Big]$

$D$ = replay buffer; $\theta$ = online network params; $\theta^-$ = target network params; $\gamma$ = discount factor. (2 marks: 1 for loss form, 1 for symbol definitions)
Target network reasoning: the bootstrap target depends on the network's own predictions; if the target used $\theta$ it would shift every gradient step, causing a "moving target" and unstable/diverging training. Freezing $\theta^-$ (updated periodically) stabilizes the target. (2 marks)

Q2. (3 marks)

Purpose: store past transitions $(s,a,r,s')$ in a buffer and sample mini-batches randomly for updates. (1 mark)
Benefits (any two, 1 each): (i) breaks temporal correlation between consecutive samples → data closer to i.i.d.; (ii) improves data efficiency by reusing transitions many times; (iii) smooths the training distribution / reduces variance. (2 marks)

Q3. (4 marks) Double DQN target: $y = r + \gamma\, Q\big(s',\ \arg\max_{a'} Q(s',a';\theta);\ \theta^-\big)$ i.e. the online network selects the action, the target network evaluates it. (3 marks)

Fixes the overestimation bias of the $\max$ operator caused by using the same network to both select and evaluate the action. (1 mark)

Q4. (4 marks)

Architecture: a shared feature backbone splits into two streams — a scalar value stream $V(s;\theta,\beta)$ and a per-action advantage stream $A(s,a;\theta,\alpha)$ . (1.5 marks)
Aggregation: $Q(s,a) = V(s) + \Big(A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a')\Big)$ (1.5 marks)
Mean subtraction is needed for identifiability: $V$ and $A$ are unidentifiable up to a constant; forcing the advantages to have zero mean makes the decomposition unique and stabilizes learning. (1 mark)

Q5. (4 marks) REINFORCE update: $\theta \leftarrow \theta + \alpha\, \nabla_\theta \log \pi_\theta(a_t|s_t)\, G_t$ where $G_t = \sum_{k=t}^{T}\gamma^{k-t} r_{k+1}$ . (3 marks)

REINFORCE replaces $Q^\pi(s,a)$ with the Monte-Carlo return $G_t$ (an unbiased sample estimate of $Q^\pi$ ). (1 mark)

Q6. (4 marks)

Advantage: $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ . (2 marks)
Variance reduction: the critic estimates $V^\pi(s)$ as a baseline, so the actor is updated using $A^\pi$ (or a TD error) rather than the raw high-variance return $G_t$ ; subtracting the state-dependent baseline lowers gradient variance without adding bias. (2 marks)

Q7. (5 marks) $L^{CLIP}(\theta) = \mathbb{E}_t\Big[\min\big(r_t(\theta)\hat{A}_t,\ \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\big)\Big]$

$r_t(\theta) = \dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ = probability ratio; $\epsilon$ = clip range (e.g. 0.2); $\hat A_t$ = advantage estimate. (3 marks)
Clipping preferred because it enforces a "trust region" with only first-order updates (simple SGD), avoiding TRPO's expensive second-order/conjugate-gradient computation of the KL constraint. (2 marks)

Q8. (4 marks) SAC objective: $J(\pi) = \sum_t \mathbb{E}_{(s_t,a_t)\sim\rho_\pi}\big[r(s_t,a_t) + \alpha\, \mathcal{H}(\pi(\cdot|s_t))\big]$ where $\mathcal{H}$ is the policy entropy and $\alpha$ the temperature. (3 marks)

Benefit: the entropy bonus encourages exploration and robustness, preventing premature convergence to deterministic policies. (1 mark)

Q9. (4 marks) $G_1 = r_1 + \gamma r_2 + \gamma^2 r_3 = 1 + 0.9(0) + 0.81(2)$ $= 1 + 0 + 1.62 = 2.62$ (Working: 2 marks; correct value 2.62: 2 marks)

Q10. (4 marks)

Model-free: learns a policy/value function directly from experience without modeling the environment (e.g. DQN or PPO). (1 mark)
Model-based: learns/uses a model of transition dynamics $P(s'|s,a)$ and reward to plan or generate imagined data (e.g. Dyna-Q or MBPO). (1 mark)
Reward shaping: adds an auxiliary shaping reward (ideally potential-based $F=\gamma\Phi(s')-\Phi(s)$ ) to give denser feedback and guide the agent toward goals in sparse-reward tasks without changing the optimal policy. (2 marks)

[
  {"claim":"Discounted return G1 = 1 + 0.9*0 + 0.81*2 = 2.62","code":"r1,r2,r3,g=1,0,2,0.9; G1=r1+g*r2+g**2*r3; result = abs(G1-2.62)<1e-9"},
  {"claim":"Dueling mean-subtraction preserves Q sum invariance: subtracting mean of advantages leaves aggregation consistent","code":"import sympy as sp; A=[sp.Symbol('a%d'%i) for i in range(3)]; V=sp.Symbol('V'); mean=sum(A)/3; Q=[V+(a-mean) for a in A]; result = sp.simplify(sum(Q) - (3*V + (sum(A)-3*mean)))==0"},
  {"claim":"Double DQN action selection by online net differs structurally from vanilla max (checked via symbolic distinctness of two Q tables)","code":"import sympy as sp; qon=[sp.Integer(1),sp.Integer(3)]; qtar=[sp.Integer(5),sp.Integer(2)]; a_star=0 if qon[0]>qon[1] else 1; ddqn=qtar[a_star]; vanilla=max(qtar); result = (ddqn==2) and (vanilla==5) and (ddqn!=vanilla)"},
  {"claim":"PPO clip bounds ratio into [1-eps,1+eps] for eps=0.2","code":"eps=0.2; r=1.5; clipped=min(max(r,1-eps),1+eps); result = abs(clipped-1.2)<1e-9"}
]