Question bank — Proximal Policy Optimization (PPO)

True — the min can only pick the smaller of the two branches, so $L^{CLIP}$ is a pessimistic lower bound on $L^{CPI}=r_t{\hat{A}}_t$; it never overstates the improvement.

False — clipping bounds the per-sample ratio, not the aggregate KL Divergence; the total KL can still grow across many samples and epochs, which is why implementations often add early-stopping on measured KL as a safety net.

True — the objective is $r_t{\hat{A}}_t$-based, so ${\hat{A}}_t=0$ zeroes both branches and the policy gradient; only the value and entropy terms could still move parameters.

Partly — the min combined with the clip removes the incentive to move further when doing so would help, but it does not hard-stop motion; a single mis-estimated batch with active gradients elsewhere can still shift the policy.

False — PPO gets TRPO-like reliability using only first-order SGD; it replaces TRPO's Fisher-matrix / conjugate-gradient machinery with a cheap clipped objective.

False — $r_t=1$ means the ratio is unchanged, but the gradient ${\nabla }_{\theta }{r}_t={\nabla }_{\theta }\log {\pi }_{\theta }$ is generally nonzero, so this sample still pushes the policy in the direction of its advantage.

False — a larger $ϵ$ permits bigger steps but reopens the instability PPO exists to prevent; too large and the importance ratio stops approximating the true objective, exactly the failure mode of unclipped importance sampling.

False — more epochs squeeze more learning from data (the win over REINFORCE), but each epoch drives ${\pi }_{\theta }$ further from ${\pi }_{{\theta }_{old}}$, so ratios drift outside the band and later epochs contribute mostly clipped, gradient-free samples.

Error — clip being active is necessary but not sufficient; the gradient is zero only when the min selects the clipped branch. When $\hat{A}<0$ and $r_t=1.3$, the ratio is clipped, yet the min keeps the unclipped branch (it is more negative), so the gradient still flows.

Error — the surrogate is only trustworthy near ${\pi }_{{\theta }_{old}}$; pushing $r_t\to \infty$ exploits a possibly noisy $\hat{A}$ and leaves the region where the reweighting is valid. PPO caps the reward at $(1+ϵ){\hat{A}}_t$ precisely to kill this incentive.

Error — the data was generated by ${\pi }_{{\theta }_{old}}$, not ${\pi }_{\theta }$, so you must reweight by the ratio $r_t$ (Importance Sampling); averaging without $r_t$ estimates the wrong expectation.

Error — we minimize squared value error between the critic $V_{\theta }$ and its target $V^{target}$; it is subtracted from $L$ so that maximizing $L$ minimizes that error. The sign is bookkeeping, not a claim that large value error is good.

Error — entropy $H=-{\sum }_a\pi \log \pi$ is added to encourage exploration, keeping the policy from collapsing to a deterministic choice too early; it is about behaviour, not numerical smoothness.

Error — $\lambda =1$ gives the unbiased Monte-Carlo return but with high variance; GAE deliberately uses $\lambda \approx 0.95$ to trade a little bias for a large variance reduction, which trains more stably.

Error — improving a valid lower bound on the surrogate still drives the true objective up as long as the bound is tight near ${\pi }_{{\theta }_{old}}$; the pessimism only prevents overclaiming improvement, not real improvement.

Because importance sampling reweights expectations by a ratio of densities — it is the exact factor that converts an average under ${\pi }_{{\theta }_{old}}$ into one under ${\pi }_{\theta }$; a difference has no such correction meaning.

The danger is the policy drifting too far, which is measured by how much the ratio moved; clipping ${\hat{A}}_t$ would distort the learning signal itself, while clipping $r_t$ only limits the step size implied by each sample.

The min is deliberately asymmetric: it lets the objective get worse freely (so you can always correct a mistake, even outside the band) but forbids it getting artificially better (so you can't be rewarded for over-stepping). Symmetric clipping would also cap useful corrections.

Because at every sample it takes the smaller of the two estimates, so it never assumes the update helped more than the conservative branch allows — it errs toward underclaiming improvement.

The clip approximates the trust region with a first-order penalty, delivering ~90% of TRPO's reliability at ~10% of the complexity; keeping both would mostly restore the expensive second-order machinery PPO was designed to avoid.

To extract more signal from each expensive rollout — the sample-efficiency gain over REINFORCE's one-step-per-batch — with the clip acting as the guardrail that keeps those extra epochs from wandering too far.

Both branches give $(1+ϵ){\hat{A}}_t$, so the value is identical; the gradient is defined by the branch structure but any further increase in $r_t$ crosses into the flat, zero-gradient region.

The clipped branch $0.8{\hat{A}}_t$ is smaller (more negative) than the unclipped $0.5{\hat{A}}_t$, so min selects it → gradient zero; PPO refuses to suppress an already-crushed bad action even further in one update.

Unclipped $1.3{\hat{A}}_t$ is more negative than clipped $1.2{\hat{A}}_t$, so min keeps the unclipped term → gradient flows to push $r_t$ down. This is the min letting us undo damage even outside the trust band.

The ratio collapses toward zero; for $\hat{A}>0$ this heavily penalizes abandoning a good action, but numerically it can blow up in log-space, which is why PPO clips and why probabilities are floored in practice.

A tiny denominator makes the ratio explode — the classic high-variance failure of Importance Sampling; clipping caps the reward from such samples, but they remain a warning that the two policies are drifting apart.

Exploration stops: every rollout samples the same action, advantages stop being informative for alternatives, and learning stalls — exactly why the entropy term is kept nonzero early in training.

The band collapses to the single point $r_t=1$; any deviation is immediately clipped and, for helpful directions, gradient-free — the policy can barely move, so learning nearly freezes.

Their raw scale multiplies directly into $r_t{\hat{A}}_t$, so a few large-magnitude advantages can dominate the update; standardizing ${\hat{A}}_t$ per batch (see the normalization definition above) keeps the band $\pm ϵ$ meaningful.

Answer