6.4.10 · D5AI Safety & Alignment
Question bank — Privacy (differential privacy, membership inference)
This page hunts the misconceptions that make differential privacy and membership inference feel slippery. Every item is a one-line reveal: read the prompt, commit to an answer in your head, then reveal. The answer side always tells you why, not just whether.
Before you start, three words we lean on constantly (all built in the parent note):
True or false — justify
DP guarantees an attacker can never learn anything about an individual.
False. DP bounds how much your specific record changes the output; an attacker can still learn population-level facts that happen to apply to you (e.g. "smokers get cancer more"), which is not a privacy violation.
A mechanism that always outputs the constant satisfies -DP for every .
True. The output distribution is identical for all datasets, so the ratio ; it is perfectly private (and perfectly useless).
Setting gives the strongest possible privacy and is what we should always aim for.
True that it is strongest ( forces identical output distributions), but false as a goal: means the output can't depend on the data at all, so no useful learning is possible.
Larger means more privacy.
False, and it's the classic sign flip. Large makes large, allowing the two neighboring output distributions to differ a lot — that is weaker privacy.
Under the replace-one neighboring definition, the sensitivity of a sum query is the same as under add/remove.
False. Replace-one removes one record and adds another, so a sum can shift by two records' worth — roughly double the add/remove sensitivity. You must always state which definition you use.
Gaussian noise gives you pure -DP just like Laplace noise.
False. Gaussian tails decay like , not , so the log-ratio isn't bounded for all outputs; Gaussian gives approximate -DP, where is a small chance the guarantee fails.
If a model is trained with DP-SGD, running membership inference on it is pointless because the attack will always fail.
False. DP bounds the attacker's advantage; it doesn't drop it to zero. With a loose the attack can still succeed noticeably — DP guarantees a limit, not immunity.
Privacy loss over training steps grows linearly, so training twice as long doubles .
False. Advanced composition makes it grow like , so training twice as long multiplies by only about , not .
Clipping gradients to norm is done to speed up convergence.
False (that's a side effect at best). Clipping bounds the sensitivity of the summed gradient to , which is what lets us calibrate a finite amount of noise — without it the sensitivity, and hence required noise, is unbounded.
Subsampling a mini-batch hurts privacy because you touch more of the data.
False. It amplifies privacy: any individual appears in only a fraction of steps, so on most steps their data isn't even used, and scales down with .
Spot the error
"Sensitivity of the mean age (ages in , , add/remove) is because that's the max age."
The error is forgetting the division by . Adding one person shifts the sum by up to , but the mean by up to ; sensitivity is measured on the quantity you release.
"To make the release more private, I added Laplace noise with scale ."
The formula is inverted. Correct scale is — bigger (weaker privacy) should give less noise, and dividing achieves that.
"I clipped each gradient to , then added to each individual gradient before summing."
Wrong point of injection. Noise is added once, to the summed (or averaged) gradient, calibrated to the sum's sensitivity . Noising every gradient separately injects times too much noise.
"Since the moments-accountant formula has no in it, clipping norm doesn't affect privacy."
Subtle trap. is absorbed into , the noise multiplier (ratio of noise std to sensitivity). Privacy depends on that ratio, so matters — it just doesn't appear as a separate free knob.
"DP protects the training data, so I can safely publish the raw dataset alongside the DP model."
The error defeats the whole point. DP protects data by not exposing it; publishing the raw records is a total privacy breach regardless of how private the model is.
"A membership inference attacker needs access to the model's internal weights to succeed."
False premise. Black-box attacks work using only query outputs (e.g. the confidence/loss on a sample), because overfit models are systematically more confident on their training examples.
"-DP with and records is fine."
Dangerous. is a per-run failure probability; the rule of thumb is (here or smaller), because a near can "leak one whole record" with meaningful probability.
Why questions
Why do we take the ratio of output probabilities rather than their difference in the DP definition?
A ratio is scale-free and multiplicative, which makes add up under composition of mechanisms; a difference wouldn't compose cleanly and would depend on the size of the probabilities.
Why does the Laplace mechanism use tails specifically?
The absolute value in the exponent means the log-density is piecewise linear, so the log-ratio of two shifted Laplace densities is bounded by (shift)/ — exactly the finite bound -DP demands.
Why must the noise scale grow with sensitivity ?
Sensitivity is how much one person can move the true answer; to hide that movement in the noise, the noise must be at least as large — otherwise the shift would poke through and reveal the person.
Why is composition (the fact that repeated queries erode privacy) the central practical worry?
Every gradient step, every released statistic, is another look at the data; each look spends budget, and accumulates, so a "private" mechanism run thousands of times can end up not private at all.
Why does overfitting make membership inference easier?
An overfit model treats training points specially — lower loss, higher confidence — so the attacker just checks "is this sample suspiciously well-predicted?" The gap between train and test behavior is the leak.
Why does DP defend against membership inference at all?
Membership inference asks "is this record in ?" — precisely the question DP bounds, since neighboring datasets (with vs. without that record) must produce nearly indistinguishable outputs.
Edge cases
What is the sensitivity of a "count how many records satisfy property P" query under add/remove?
Exactly : adding or removing one person can change the count by at most one, regardless of dataset size.
What happens to the Laplace noise scale as ?
: the noise becomes infinitely large, drowning the signal completely — the price of demanding perfect (indistinguishable) privacy.
If two datasets differ in two records, does -DP say anything about them?
Yes, by composition/group privacy: differing in records gives a bound of , so two-record differences are bounded by — weaker, but not silent.
A gradient already has norm . What does clipping do to it?
Nothing. The factor , so small gradients pass through unchanged; only gradients exceeding are shrunk.
Batch size (full-batch, no subsampling). What is the amplification rate ?
, so there is no privacy amplification — every record participates in every step, which is the worst case for the subsampling bound.
If a mechanism is deterministic (adds no randomness), can it be -DP for finite ?
Only if its output ignores the differing record entirely. Any deterministic mechanism whose output actually changes between neighbors has some output with probability on one and on the other, forcing an infinite ratio.
What does in -DP represent at the extreme ?
It recovers pure -DP: zero probability of the guarantee failing, which is exactly what the Laplace mechanism delivers and Gaussian cannot.
Recall Self-check
Larger means ______ privacy. ::: weaker Laplace noise scale is ______ . ::: divided by Privacy loss over steps grows like ______. ::: Membership inference is easier when a model is ______. ::: overfit Subsampling rate ______ privacy. ::: amplifies (improves)