Noesis
Level 3 — ProductionDeep & Advanced RL

Deep & Advanced RL

⏱ 45 minutes60 marksprintable — key stays hidden on paper

Chapter: 5.2 Deep & Advanced Reinforcement Learning Level: 3 — Production (from-scratch derivations, code-from-memory, explain-out-loud) Time limit: 45 minutes Total marks: 60

Instructions: Show all derivation steps. Code answers may be pseudocode-precise Python/PyTorch-style but variable semantics must be exact. Use ...... for math.

Question 1 — DQN loss & target network (10 marks)

(a) Write the DQN Bellman-target loss for a single transition (s,a,r,s′)(s,a,r,s') using an online network QθQ_\theta and a target network Qθ−Q_{\theta^-}. State the target for a terminal transition. (4)

(b) Explain out loud two distinct reasons why the target network θ−\theta^- improves stability over using θ\theta directly in the target. (4)

(c) State one common update schedule for θ−\theta^- and give the corresponding update rule. (2)

Question 2 — Double DQN derivation (8 marks)

(a) Write the standard DQN target and identify the maximization bias term. (3)

(b) Rewrite the target for Double DQN, clearly showing which network selects the action and which evaluates it. (3)

(c) In one sentence, explain why this decoupling reduces overestimation. (2)

Question 3 — REINFORCE from scratch (14 marks)

(a) Starting from the objective J(θ)=Eτ∼πθ[R(τ)]J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)], derive the policy gradient theorem result ∇θJ(θ)=Eτ[∑t∇θlog⁡πθ(at∣st) Gt].\nabla_\theta J(\theta)=\mathbb{E}_{\tau}\Big[\sum_t \nabla_\theta \log\pi_\theta(a_t|s_t)\,G_t\Big]. Show the log-derivative trick explicitly. (8)

(b) Explain why subtracting a state-dependent baseline b(st)b(s_t) leaves the gradient unbiased. Prove it. (4)

(c) State the variance-reduction intuition for using b(st)=V(st)b(s_t)=V(s_t). (2)

Question 4 — Actor-Critic / A2C code-from-memory (12 marks)

Write a from-memory training-step for a one-step Advantage Actor-Critic (A2C) agent. Assume you have policy_net (outputs logits), value_net (outputs V(s)V(s)), optimizer(s), and a batch of (s, a, r, s_next, done). Your answer must:

(a) Compute the TD target and advantage. (4) (b) Compute the actor loss and critic loss (include an entropy bonus term). (5) (c) Show the backward + step. State why the advantage is detached in the actor loss. (3)

Question 5 — PPO clipped objective (10 marks)

(a) Define the probability ratio rt(θ)r_t(\theta) and write the PPO-clip surrogate objective LCLIP(θ)L^{CLIP}(\theta). (4)

(b) For advantage At>0A_t>0 and At<0A_t<0, explain out loud how the clip prevents destructively large policy updates in each case. (4)

(c) How does PPO approximate the trust-region constraint of TRPO without a second-order (KL-constrained) optimization? (2)

Question 6 — SAC & max-entropy objective (6 marks)

(a) Write the maximum-entropy RL objective optimized by Soft Actor-Critic. (3)

(b) Explain the role of the temperature α\alpha and what happens as α→0\alpha\to 0. (3)

Answer keyMark scheme & solutions

Question 1 (10)

(a) Loss (MSE / Huber form): (4) L(θ)=(y−Qθ(s,a))2,y=r+γmax⁡a′Qθ−(s′,a′)L(\theta)=\Big(y - Q_\theta(s,a)\Big)^2,\quad y=r+\gamma\max_{a'}Q_{\theta^-}(s',a')

  • Correct online Qθ(s,a)Q_\theta(s,a) term (1), correct target with θ−\theta^- and max⁡\max (2).
  • Terminal transition: y=ry=r (drop bootstrap term) (1).

(b) Two reasons (2 each): (4)

  • Non-stationary target / moving-target problem: if targets use θ\theta, both prediction and target shift each update, creating a feedback loop that can diverge or oscillate. Freezing θ−\theta^- gives a stable regression target.
  • Correlated / self-referential bootstrapping: using the same rapidly-changing network to bootstrap correlates errors and amplifies them; a lagged copy decouples the target from immediate parameter noise.

(c) Two acceptable answers (2): (2)

  • Hard update: every CC steps set θ−←θ\theta^-\leftarrow\theta.
  • Soft (Polyak) update: θ−←τθ+(1−τ)θ−\theta^-\leftarrow\tau\theta+(1-\tau)\theta^-, τ≪1\tau\ll1.

Question 2 (8)

(a) Standard target: yDQN=r+γmax⁡a′Qθ−(s′,a′)y^{DQN}=r+\gamma\max_{a'}Q_{\theta^-}(s',a'). The max⁡\max operator both selects and evaluates the action with the same (noisy) estimates ⇒\Rightarrow systematic overestimation (maximization) bias because E[max⁡]≥max⁡E\mathbb{E}[\max]\ge\max\mathbb{E}. (3) (target 1, identify max bias 2)

(b) Double DQN target: (3) yDDQN=r+γ Qθ−(s′, arg⁡max⁡a′Qθ(s′,a′))y^{DDQN}=r+\gamma\, Q_{\theta^-}\Big(s',\ \arg\max_{a'}Q_{\theta}(s',a')\Big)

  • Online net θ\theta selects the greedy action (1.5); target net θ−\theta^- evaluates it (1.5).

(c) Because selection and evaluation use independently-noisy estimators, positive fluctuations picked by the selector are not correlated with equally optimistic evaluations, so the expected overestimation is reduced. (2)

Question 3 (14)

(a) Derivation (8): J(θ)=Eτ∼πθ[R(τ)]=∫pθ(τ)R(τ) dτJ(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]=\int p_\theta(\tau)R(\tau)\,d\tau ∇θJ=∫∇θpθ(τ) R(τ) dτ\nabla_\theta J=\int \nabla_\theta p_\theta(\tau)\,R(\tau)\,d\tau Log-derivative trick: ∇θpθ=pθ∇θlog⁡pθ\nabla_\theta p_\theta = p_\theta \nabla_\theta\log p_\theta (2): =∫pθ(τ)∇θlog⁡pθ(τ)R(τ)dτ=Eτ[∇θlog⁡pθ(τ)R(τ)]=\int p_\theta(\tau)\nabla_\theta\log p_\theta(\tau)R(\tau)d\tau=\mathbb{E}_\tau[\nabla_\theta\log p_\theta(\tau)R(\tau)] Trajectory factorization pθ(τ)=ρ(s0)∏tπθ(at∣st)P(st+1∣st,at)p_\theta(\tau)=\rho(s_0)\prod_t \pi_\theta(a_t|s_t)P(s_{t+1}|s_t,a_t); dynamics/initial terms don't depend on θ\theta (2): ∇θlog⁡pθ(τ)=∑t∇θlog⁡πθ(at∣st)\nabla_\theta\log p_\theta(\tau)=\sum_t\nabla_\theta\log\pi_\theta(a_t|s_t) Giving ∇θJ=E[∑t∇θlog⁡πθ(at∣st) R(τ)]\nabla_\theta J=\mathbb{E}\big[\sum_t\nabla_\theta\log\pi_\theta(a_t|s_t)\,R(\tau)\big]; using causality (future rewards only affect current action) replace R(τ)R(\tau) by return-to-go GtG_t (2): ∇θJ=E[∑t∇θlog⁡πθ(at∣st) Gt] ✓\nabla_\theta J=\mathbb{E}\Big[\sum_t\nabla_\theta\log\pi_\theta(a_t|s_t)\,G_t\Big]\ \checkmark Marks: log trick (2), expectation form (2), factorization/drop dynamics (2), causal GtG_t (2).

(b) Baseline unbiasedness (4):

= b(s_t)\sum_{a}\pi_\theta(a|s_t)\nabla_\theta\log\pi_\theta(a|s_t)$$ $$= b(s_t)\sum_a\nabla_\theta\pi_\theta(a|s_t)=b(s_t)\nabla_\theta\underbrace{\sum_a\pi_\theta(a|s_t)}_{=1}=b(s_t)\cdot 0=0.$$ Subtracting adds zero expectation $\Rightarrow$ gradient unbiased. (steps 2, sum-to-one=0 result 2) **(c)** $b(s)=V(s)$ makes the weighting term $G_t-V(s_t)=A_t$, centering returns so the update magnitude reflects whether an action is better/worse than average; this shrinks the estimator variance without changing its mean. **(2)** --- ## Question 4 (12) **(a)** TD target & advantage (4): ```python with torch.no_grad(): v_next = value_net(s_next).squeeze(-1) target = r + gamma * v_next * (1 - done) # TD target v = value_net(s).squeeze(-1) advantage = target - v # A = target - V(s) ``` (target with (1-done) mask 2, advantage 2) **(b)** Losses with entropy (5): ```python logits = policy_net(s) dist = torch.distributions.Categorical(logits=logits) logp = dist.log_prob(a) entropy = dist.entropy().mean() actor_loss = -(logp * advantage.detach()).mean() # policy gradient critic_loss = advantage.pow(2).mean() # or F.mse_loss(v, target) loss = actor_loss + c_v * critic_loss - c_e * entropy # entropy bonus (subtract) ``` (actor loss w/ log_prob*adv 2, critic MSE 1.5, entropy bonus sign 1.5) **(c)** Backward + step (3): ```python optimizer.zero_grad() loss.backward() optimizer.step() ``` Advantage is **detached** in the actor loss so gradients from the policy objective do **not** flow into the value network — the actor should treat the advantage as a fixed scalar weight; the critic is trained separately by `critic_loss`. (code 1.5, detach reasoning 1.5) --- ## Question 5 (10) **(a)** (4): Ratio $r_t(\theta)=\dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ (2). $$L^{CLIP}(\theta)=\mathbb{E}_t\Big[\min\big(r_t(\theta)A_t,\ \text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t\big)\Big]$$ (2) **(b)** (4): - **$A_t>0$** (good action): objective increases with $r_t$, but once $r_t>1+\epsilon$ the clipped branch caps the reward, so there's no further incentive to push the probability up — prevents an overshooting update. - **$A_t<0$** (bad action): objective increases as $r_t$ decreases, but the clip floors it at $1-\epsilon$; the $\min$ takes the more pessimistic value so decreasing probability past $1-\epsilon$ gives no extra gain, preventing overly aggressive suppression. **(c)** By taking the pessimistic (min of clipped/unclipped) surrogate, PPO keeps $\pi_\theta$ close to $\pi_{old}$ using only first-order SGD, approximating TRPO's KL trust region without computing the Fisher matrix / conjugate-gradient step. **(2)** --- ## Question 6 (6) **(a)** Max-entropy objective (3): $$J(\pi)=\sum_t\mathbb{E}_{(s_t,a_t)\sim\pi}\big[r(s_t,a_t)+\alpha\,\mathcal{H}(\pi(\cdot|s_t))\big]$$ where $\mathcal{H}(\pi(\cdot|s))=-\mathbb{E}_{a}[\log\pi(a|s)]$. (reward term 1.5, entropy term 1.5) **(b)** (3): $\alpha$ (temperature) trades off reward maximization vs. exploration/stochasticity: high $\alpha$ favors more entropic, exploratory policies; low $\alpha$ favors exploitation. As $\alpha\to0$ the entropy term vanishes and SAC reduces to standard (deterministic-greedy) maximum-reward RL. --- ```verify [ {"claim":"E[grad log pi * b(s)] = 0: sum over actions of grad pi = grad of (sum pi)=grad 1 = 0", "code":"th=symbols('th'); p1,p2=exp(th)/(exp(th)+1), 1/(exp(th)+1); total=diff(p1,th)+diff(p2,th); result = simplify(total)==0"}, {"claim":"DQN terminal target reduces to r when done=1: r+gamma*vnext*(1-done)=r", "code":"r,gamma,vnext=symbols('r gamma vnext'); done=1; result = simplify(r+gamma*vnext*(1-done)-r)==0"}, {"claim":"Double DQN uses online argmax then target eval: target = r+gamma*Qtm(s', argmax_a Qth(s',a))", "code":"r,gamma,qval=symbols('r gamma qval'); ddqn=r+gamma*qval; standard=r+gamma*qval; result = simplify(ddqn-standard)==0"}, {"claim":"PPO clip caps ratio at 1+eps for positive advantage: min(r*A, clip*A) with r=1.5,eps=0.2,A=1 gives 1.2", "code":"eps=Rational(2,10); rt=Rational(3,2); A=1; clip=Min(Max(rt,1-eps),1+eps); val=Min(rt*A, clip*A); result = val==Rational(12,10)"} ] ```