Privacy (differential privacy, membership inference)

Overview

Machine learning models can memorize training data, creating serious privacy risks. Two key concepts protect against this:

Differential Privacy (DP): A mathematical framework guaranteeing that any individual's data has minimal impact on model outputs
Membership Inference Attacks: Adversarial techniques that determine whether a specific record was in the training set

These form complementary perspectives: DP is the defense, membership inference is the attack that motivates the defense.

Differential Privacy: The Mathematical Defense

Deriving the Privacy Guarantee

Step 1: What does "neighboring datasets" mean?

There are two standard definitions of "neighboring," and they change the sensitivity by a factor of 2:

Add/Remove definition: $D_2$ is $D_1$ with one record added or removed. $|D_1| \neq |D_2|$ .
Replace-one definition: $D_2$ is $D_1$ with one record replaced by another. $|D_1| = |D_2|$ .

Example (add/remove):

$D_1 = \{x_1, x_2, \ldots, x_n\}$
$D_2 = \{x_1, x_2, \ldots, x_n, x_{n+1}\}$ (one extra person)

Why this matters: Under add/remove, a summed query changes by at most one record's contribution. Under replace-one, it can change by two record's worth (remove old + add new), doubling the sensitivity. We state which definition we use in every derivation below.

Step 2: Why the exponential bound?

Take the ratio of probabilities:

$\frac{\Pr[\mathcal{M}(D_1) \in S]}{\Pr[\mathcal{M}(D_2) \in S]} \leq e^{\varepsilon}$

Why this form? The exponential makes ε additive under composition (we'll see why this matters). For small ε, $e^{\varepsilon} \approx 1 + \varepsilon$ , meaning probabilities are "close."

Step 3: What's a "mechanism"?

A mechanism $\mathcal{M}$ is any algorithm that takes data and produces output. For ML:

Computing gradients
Releasing model parameters
Answering queries about data statistics

The guarantee: An adversary with unlimited computational power, seeing $\mathcal{M}(D)$ , cannot confidently determine whether any specific individual is in $D$ .

Formula The Laplace Mechanism

To release a numeric query $f(D)$ with ε-DP, add noise from the Laplace distribution: $\mathcal{M}(D) = f(D) + \text{Lap}\left(\frac{\Delta f}{\varepsilon}\right)$ Where:

$\Delta f = \max_{D_1, D_2 \text{ neighbors}} |f(D_1) - f(D_2)|$ is the sensitivity
$\text{Lap}(b)$ has PDF $p(x) = \frac{1}{2b}e^{-|x|/b}$

Derivation: For neighbors $D_1, D_2$ , let $\delta = f(D_1) - f(D_2)$ , where $|\delta| \leq \Delta f$ .

$\frac{\Pr[\mathcal{M}(D_1) = z]}{\Pr[\mathcal{M}(D_2) = z]} = \frac{e^{-|z - f(D_1)|/b}}{e^{-|z - f(D_2)|/b}} = e^{(|z - f(D_2)| - |z - f(D_1)|)/b}$

By triangle inequality, the numerator $\leq |\delta| \leq \Delta f$ . Setting $b = \Delta f/\varepsilon$ gives:

$\frac{\Pr[\mathcal{M}(D_1) = z]}{\Pr[\mathcal{M}(D_2) = z]} \leq e^{\varepsilon}$

Why this step? The Laplace distribution's exponential tails create the exact bound we need. Gaussian noise doesn't satisfy pure DP (though it satisfies approximate DP).

DP-SGD: Differential Privacy in Deep Learning

Standard SGD computes:

$\theta_{t+1} = \theta_t - \eta \cdot \frac{1}{B}\sum_{i=1}^B \nabla_\theta \ell(\theta_t, x_i)$

Problem: Each gradient $\nabla \ell(\theta, x_i)$ can contain unbounded information about individual $x_i$ .

DP-SGD Fix (Abadi et al., 2016):

Clip each gradient: $\bar{g}_i = g_i / \max(1, \|g_i\|_2 / C)$
- Why? Bounds each per-example gradient norm to $C$ . Under the add/remove neighboring definition, removing one clipped gradient from the sum $\sum_i \bar{g}_i$ changes it by at most $C$ , so the sensitivity of the summed gradient is $\Delta = C$ . (Under the replace-one definition it would be $2C$ , since you remove one contribution of norm $\leq C$ and add another.) We use add/remove throughout.
Add Gaussian noise: To the summed clipped gradient $\sum_i \bar{g}_i$ we add noise with standard deviation proportional to the sensitivity $C$ : $\tilde{g} = \frac{1}{B}\left(\sum_{i=1}^B \bar{g}_i + \mathcal{N}(0, \sigma^2 C^2 I)\right)$
- Why the $1/B$ outside? Noise is calibrated to the sensitivity of the sum ( $\sim \sigma C$ ), and then the whole sum (signal + noise) is averaged by $1/B$ . Equivalently, the noise added to the averaged gradient has standard deviation $\sigma C / B$ per coordinate. Both statements describe the same mechanism; the noise is scaled to sensitivity $C$ of the sum before averaging.
- Why Gaussian? We use (ε, δ)-DP (approximate DP) where small failure probability δ is acceptable.
Update: $\theta_{t+1} = \theta_t - \eta \tilde{g}$

Privacy accounting: After $T$ steps with Poisson subsampling rate $q = B/N$ , privacy degrades. The moments accountant / advanced-composition bound for the subsampled Gaussian mechanism is:

$\varepsilon(T) \;\approx\; q \,\sqrt{2\,T \ln(1/\delta)} \,\Big/\, \sigma$

Why this scaling? Privacy loss accumulates as $\sqrt{T}$ (not linearly!) due to advanced composition, and is amplified by subsampling through the factor $q = B/N$ : an individual only appears in a $q$ fraction of steps. Note the sensitivity $C$ and batch size $B$ do not appear separately here — they are absorbed into the noise multiplier $\sigma$ (the ratio of noise std to sensitivity), which is the quantity that actually controls privacy.

Worked example Example 1: DP Mean Release

Task: Release the mean age of $n=1000$ patients. Ages $\in [0, 100]$ . Use the add/remove definition.

Step 1: Compute true mean $\mu = \frac{1}{n}\sum_{i=1}^n \text{age}_i$

Step 2: Calculate sensitivity

Adding/removing one person changes the sum by at most 100
Mean changes by at most $\Delta f = 100/1000 = 0.1$ Why this value? Worst case: a single record of age 100 is added or removed, shifting the sum by 100.

Step 3: Add noise for ε = 1.0 $\tilde{\mu} = \mu + \text{Lap}\!\left(\frac{\Delta f}{\varepsilon}\right) = \mu + \text{Lap}\!\left(\frac{0.1}{1.0}\right) = \mu + \text{Lap}(0.1)$

Step 4: Interpret

50% of noise is within $\pm 0.069$ (median absolute deviation of Laplace with $b=0.1$ )
95% within $\pm 0.3$ Why this matters: For population statistics, DP adds negligible error but strong privacy.

Worked example Example 2: DP-SGD for Binary Classifier

Setup: Train logistic regression on $N=1000$ records with DP (add/remove definition).

Parameters:

Batch size $B = 250$ → subsampling rate $q = B/N = 0.25$
Clipping norm $C = 1.0$
Noise multiplier $\sigma = 4.0$
Epochs $E = 10$ → $T = 400$ steps
Target $\delta = 10^{-5}$

Step 1: Per-example gradient clipping

for x_i in batch:
   g_i = compute_gradient(x_i)
   g_i = g_i / max(1.0, ||g_i|| / C)   # bound norm to C

Why clip? Unbounded gradients → unbounded sensitivity → unbounded privacy loss.

Step 2: Aggregate and add noise (calibrated to sensitivity C of the sum)

g_sum   = sum(clipped_gradients)
noise   = Normal(0, sigma * C)          # std ~ sigma*C on the SUM
g_private = (g_sum + noise) / B         # then average
# equivalently, noise on the AVERAGE has std sigma*C/B

Why $\sigma C$ on the sum? Sensitivity of the summed gradient is $C$ ; the noise multiplier $\sigma$ scales noise relative to that sensitivity.

Step 3: Privacy accounting Using the subsampled-Gaussian bound with $q = 0.25$ , $T = 400$ , $\sigma = 4.0$ , $\delta = 10^{-5}$ : $\varepsilon \approx q\,\frac{\sqrt{2T\ln(1/\delta)}}{\sigma} = 0.25 \cdot \frac{\sqrt{2\cdot 400 \cdot \ln(10^5)}}{4.0} \approx 0.25 \cdot \frac{\sqrt{9210}}{4.0} \approx 0.25 \cdot \frac{95.97}{4.0} \approx 6.0$

Result: roughly a $(6, 10^{-5})$ -DP model under this (loose) analytic bound. In practice a numerically tight moments/RDP accountant would report a smaller ε for the same $\sigma$ ; the closed form above is an upper-bound approximation. Takeaway: the exact number depends heavily on the accountant used — always compute ε with a real accountant, never by eyeballing.

Membership Inference Attacks: The Threat

Why Models Leak Membership

Core insight: Models overfit to training data, behaving differently on train vs test examples.

Signal 1: Confidence

Training examples → high confidence predictions
Test examples → lower confidence

Why? The model has "seen" training examples many times, memorizing details.

Signal 2: Loss value

Training loss < Test loss (by definition of overfitting)
If $\ell(f, x)$ is unusually low, $x$ likely in training set

The Classic Attack (Shokri et al., 2017)

Setup:

Target model $f_{\text{target}}$ trained on $D_{\text{target}}$
Attacker has access to $f_{\text{target}}(x)$ (black-box queries)
Attacker trains shadow models on similar data

Attack Steps:

Train shadow models: $f_{\text{shadow}}^{(1)}, \ldots, f_{\text{shadow}}^{(k)}$ on datasets $D_1, \ldots, D_k$
- Why? Simulate target model's behavior to generate training data
Create attack training set:
- For each shadow model, label examples as "in" or "out" of training
- Features: prediction vector $f_{\text{shadow}}(x) = [p_1, \ldots, p_C]$
Train binary classifier: $\mathcal{A}(f(x)) \to \{0, 1\}$
- Input: prediction confidence scores
- Output: membership prediction
Apply to target: Use $\mathcal{A}(f_{\text{target}}(x))$ to infer membership

Worked example Example 3: Simple Membership Inference

Scenario: Binary classifier for medical diagnosis (diabetes), 1000 training examples.

Step 1: Compute confidence threshold

Average confidence on training set: 0.92
Average confidence on held-out validation: 0.78
Set threshold: $\tau = 0.85$ (midpoint)

Step 2: Attack rule

def is_member(model, x):
   confidence = max(model.predict_proba(x))
   return confidence > 0.85

Why this threshold? Balances false positives (claiming non-members are members) and false negatives.

Step 3: Evaluate

Test on 500 known train examples: 380 correctly identified (76% recall)
Test on 500 known test examples: 120 incorrectly identified (24% false positive rate)
AUC = 0.76

Interpretation: Significant privacy leak. An attacker can identify 3/4 of training examples.

The DP-Privacy Connection

Intuition Why DP Defends Against Membership Inference

Differential privacy directly bounds membership inference advantage. Define the attacker's true-positive rate (TPR = detect a member) and false-positive rate (FPR = flag a non-member). DP relates the two distributions (member-in vs member-out) so that:

$\text{TPR} \leq e^{\varepsilon}\cdot \text{FPR} + \delta$

and symmetrically. From this the advantage (difference of success probabilities on members vs non-members) is bounded:

$\text{Adv} \;\leq\; \frac{e^{\varepsilon}-1}{e^{\varepsilon}+1} + \delta \;\approx\; \frac{\varepsilon}{2}\ \text{(for small }\varepsilon\text{)}$

Proof sketch:

Let $D_1$ include $x$ , $D_2$ exclude $x$
Model from $D_1$ : $f_1 = \mathcal{M}(D_1)$ ; model from $D_2$ : $f_2 = \mathcal{M}(D_2)$
DP: $\Pr[\mathcal{A}(f_1)=1] \leq e^{\varepsilon}\Pr[\mathcal{A}(f_2)=1] + \delta$
Combining the two directions and simplifying gives the advantage bound above.

Why this matters: ε controls the worst-case gap between how often the attacker flags members vs non-members. This is a bound on the difference of success probabilities, not a multiplicative factor over "random guessing." For small ε the excess advantage over chance is roughly $\varepsilon/2$ .

Privacy-Utility Tradeoff

The fundamental tension:

$\text{Utility} \propto \frac{1}{\sigma^2} \propto \frac{1}{\varepsilon^2}$

Low ε (strong privacy): Need large noise → poor model accuracy
High ε (weak privacy): Less noise → better accuracy but more leakage

Practical values:

ε < 1: Strong privacy, significant utility loss
ε = 1-10: Moderate privacy, acceptable utility
ε > 10: Weak privacy, minimal utility impact

Why no free lunch? Information-theoretically, learning requires extracting signal from data. Privacy requires hiding individual contributions. These goals conflict.

Worked example Example 4: Privacy-Utility Tradeoff Analysis

Setup: Image classification (CIFAR-10), comparing privacy levels.

Privacy Budget	Accuracy	Attack AUC	Interpretation
Non-private	92%	0.82	Severe leakage
ε = 8, δ = 10⁻⁵	88%	0.64	Moderate leak
ε = 2, δ = 10⁻⁵	81%	0.53	Strong privacy
ε = 0.5, δ = 10⁻⁵	68%	0.51	Near-perfect privacy

Step-by-step interpretation:

Non-private baseline: High accuracy, but attackers can identify training data with AUC 0.82 (much better than 0.5 random).
ε = 8: Modest privacy, only 4% accuracy drop. Attack still somewhat effective (AUC 0.64).

Why still vulnerable? ε = 8 is relatively large; allows a probability ratio up to $e^8 \approx 2981$ .

ε = 2: Strong privacy guarantee. 11% accuracy drop, but attack barely better than random (0.53 vs 0.5).

Why effective? ε = 2 means the ratio bound is $e^2 \approx 7.4$ — hard for attackers to exploit.

ε = 0.5: Near-perfect privacy (attack is essentially random guessing), but 24% accuracy drop.

Why so costly? Very small ε requires very large noise relative to signal.

Takeaway: For most applications, ε ≈ 2-4 hits the sweet spot: strong privacy with manageable utility loss.

Common Mistakes

Common mistake Mistake 1: "Adding random noise = differential privacy"

Why this feels right: Noise does mask individual contributions.

Why it's wrong: DP requires calibrated noise based on sensitivity. Random noise without sensitivity analysis provides no formal guarantee.

Example:

# WRONG
noisy_mean = true_mean + random.gauss(0, 0.1)
# RIGHT
sensitivity = max_value / n
noisy_mean = true_mean + laplace(0, sensitivity / epsilon)

The fix: Always compute sensitivity first, then scale noise accordingly.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, yaha core baat samajhne ki hai — ML models sirf patterns nahi seekhte, wo apne training data ko literally memorize bhi kar lete hain. Matlab agar tumne kisi language model ko private emails pe train kiya, to wo model kisi ki private email word-by-word wapas nikaal sakta hai. Ya ek medical classifier bata sakta hai ki kisi specific person ka record training set mein tha ya nahi. Traditional privacy poochti thi "kya tum raw data dekh sakte ho?" — par ML privacy ek deeper sawaal poochti hai: "kya tum model se hi individuals ke baare mein infer kar sakte ho?" Yahi reason hai ki model khud ek privacy leak ban jaata hai, chahe tum training data kabhi na dekho.

Ab isse bachne ke liye do concepts kaam aate hain jo ek dusre ke opposite side pe khade hain. Membership Inference Attack ek adversarial technique hai jo pata lagati hai ki koi specific record training mein tha ya nahi — ye attack hai. Aur Differential Privacy (DP) uska defense hai. DP ka core idea simple intuition pe based hai: agar kisi ek bhi individual ka data add ya remove karne se model ke output pe bilkul minimal farak padta hai, to attacker kabhi confidently nahi keh paayega ki wo banda dataset mein tha ya nahi. Mathematically ye guarantee "epsilon" se control hoti hai — jitna chhota epsilon, utni strong privacy. Practically ye Laplace noise add karke achieve karte hain, jaha noise ka amount "sensitivity" (Δf) pe depend karta hai — yaani ek record maximum kitna output badal sakta hai.

Ye matter isliye karta hai kyunki aaj real-world mein healthcare, finance, aur personal data pe models train ho rahe hain, aur ek chhoti si leak legal aur ethical disaster ban sakti hai. DP ki khoobsurti ye hai ki ye ek mathematical guarantee deta hai — even agar attacker ke paas unlimited computing power ho, tab bhi wo individual ke baare mein confidently kuch nahi nikaal paayega. Isliye jab bhi tum sensitive data pe ML padhoge, ye samajhna zaroori hai ki privacy ek afterthought nahi, balki design ka core part honi chahiye — attack aur defense dono ko saath samajhna hi asli picture deta hai.