Question bank — Reward modeling

False. It outputs an unbounded scalar $r_{\theta }(x,y)\in \mathbb{R}$ (a score for response $y$ to prompt $x$); the probability only appears when we compare two scores through the sigmoid, $\sigma (r_w-r_l)$.

False. The loss and the preference depend only on the difference $\Delta =r_w-r_l$; a constant shift cancels, so rankings and gradients are unchanged.

False. It depends only on their difference: $P=\sigma (\Delta )$. Two scores of $(105,103)$ behave identically to $(5,3)$.

False for real models — a "perfect" RM is a fiction. In practice more pressure exploits RM errors (Goodhart's law), so true preference peaks then falls even as measured reward climbs.

True. Both minimize $-\log \sigma (\text{score difference})$; here the "feature" is $\Delta =r_w-r_l$ and the label is always "winner wins".

True. Loss and gradient magnitude both scale with $\sigma (-\Delta )$, which is large exactly when the model ranks the pair wrongly ($\Delta <0$).

False. Absolute ratings are noisy and mis-calibrated between annotators; scale-free pairwise comparisons are more reliable, which is why the logistic loss is standard.

False. The KL divergence term subtracts from the objective; its job is to keep the policy near the reference so the RM stays in-distribution, trading some reward for reliability.

False. DPO folds the same Bradley–Terry preference objective directly into the policy update, skipping the explicit RM and PPO loop.

The error: absolute reward has no meaning — only per-prompt relative ordering does. The needed fix is to subtract a per-prompt mean (mean-center) so scores are comparable; dividing by the standard deviation (variance scaling) is an optional convenience for RL stability, not something the Bradley–Terry math requires.

The error: a ranking of $K$ answers yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ ordered pairs. For $K=4$ that is 6 comparisons, not 4.

The error: measured reward is the proxy, not truth. Rising RM reward with falling human preference is the signature of reward hacking / over-optimization.

The error: the sigmoid asymptotes to 1 but never reaches it, so the loss $-\log \sigma (\Delta )$ is always strictly positive — there is always a residual push.

The error: the KL term is used during policy optimization, not RM training, and it needs the reference model ${\pi }_{\text{ref}}$ throughout to leash the policy.

The error: there is no target value. The loss only requires $r_w$ to exceed $r_l$ by a comfortable margin; forcing absolute magnitudes contradicts the shift-invariance of the model.

A step function has zero gradient almost everywhere, giving no learning signal and being blind to how wrong a ranking is; the sigmoid gives smooth, margin-sensitive gradients.

The factor is $\sigma (-\Delta )$; for large positive $\Delta$ it goes to zero, meaning "nothing left to fix" — the model spends effort only where ordering is wrong or shaky.

Humans can label only a tiny sample; the RM generalizes those comparisons to score the millions of on-policy samples PPO needs, which no human loop could provide.

People agree far more on "A is better than B" than on assigning a calibrated number; comparisons cancel per-annotator offsets that would otherwise become noise.

An RM is only trustworthy on text resembling its training distribution; the KL leash stops the policy from drifting into weird out-of-distribution outputs where the RM's confident scores are meaningless.

It reuses expensive human labeling and averages out the noise of any single comparison, reducing RM overfitting to individual noisy pairs (InstructGPT's trick).

$\Delta =0$, so loss $=-\log \sigma (0)=-\log (0.5)\approx 0.693$ with gradient magnitude $\sigma (0)=0.5$ — the model is still nudged to break the tie toward the labeled winner.

$\sigma (\Delta )\to 1$ so loss $\to 0$ and gradient $\to 0$; the pair is "solved" and contributes almost nothing more.

Loss $\approx |\Delta |$ grows without bound and gradient magnitude $\sigma (-\Delta )\to 1$ saturates — maximal, but bounded, correction so training stays stable.

It cannot satisfy both, so it settles near $r_w\approx r_l$ (probability $\approx 0.5$), correctly expressing that the preference is genuinely uncertain.

The RM can minimize loss by rewarding length instead of quality — a spurious correlation that later shows up as verbosity-hacking under PPO, a classic Goodhart's law trap.

Yes — equal quality means the training signal never separates them, and since only differences matter, identical scores are a valid, correct outcome.