Privacy (differential privacy, membership inference)

6.4.10 · AI-ML › AI Safety & Alignment

Overview

Machine learning models training data memorize kar sakte hain, jo serious privacy risks paida karta hai. Do key concepts isse protect karte hain:

Differential Privacy (DP): Ek mathematical framework jo guarantee karta hai ki kisi bhi individual ka data model outputs pe minimal impact kare
Membership Inference Attacks: Adversarial techniques jo determine karti hain ki koi specific record training set mein tha ya nahi

Ye dono complementary perspectives form karte hain: DP defense hai, aur membership inference woh attack hai jo is defense ko motivate karti hai.

Differential Privacy: The Mathematical Defense

Privacy Guarantee Derive Karna

Step 1: "Neighboring datasets" ka matlab kya hai?

"Neighboring" ki do standard definitions hain, aur ye sensitivity ko 2 ke factor se change kar deti hain:

Add/Remove definition: $D_{2}$ woh $D_{1}$ hai jisme ek record add ya remove kiya gaya ho. $∣ D_{1} ∣ \neq = ∣ D_{2} ∣$ .
Replace-one definition: $D_{2}$ woh $D_{1}$ hai jisme ek record replace kiya gaya ho kisi aur se. $∣ D_{1} ∣ = ∣ D_{2} ∣$ .

Example (add/remove):

$D_{1} = {x_{1}, x_{2}, \dots, x_{n}}$
$D_{2} = {x_{1}, x_{2}, \dots, x_{n}, x_{n + 1}}$ (ek extra person)

Ye kyun matter karta hai: Add/remove ke under, ek summed query zyada se zyada ek record ke contribution se change hoti hai. Replace-one ke under, ye do records ke worth se change ho sakti hai (old remove + new add), jo sensitivity double kar deta hai. Hum clearly batate hain ki hum neeche har derivation mein kaunsi definition use kar rahe hain.

Step 2: Exponential bound kyun?

Probabilities ka ratio lo:

$\frac{P r [ M ( D _{1} ) \in S ]}{P r [ M ( D _{2} ) \in S ]} \leq e^{ε}$

Ye form kyun? Exponential, ε ko composition ke under additive banata hai (hum dekhenge ki ye kyun matter karta hai). Chhote ε ke liye, $e^{ε} \approx 1 + ε$ , matlab probabilities "close" hain.

Step 3: "Mechanism" kya hota hai?

Ek mechanism $M$ koi bhi aisa algorithm hai jo data leta hai aur output produce karta hai. ML ke liye:

Gradients compute karna
Model parameters release karna
Data statistics ke baare mein queries answer karna

Guarantee: Unlimited computational power wala adversary, $M (D)$ dekhke, confidently determine nahi kar sakta ki koi specific individual $D$ mein hai ya nahi.

Ye step kyun? Laplace distribution ke exponential tails bilkul wohi bound create karte hain jo humein chahiye. Gaussian noise pure DP satisfy nahi karta (though approximate DP satisfy karta hai).

DP-SGD: Deep Learning mein Differential Privacy

Standard SGD compute karta hai:

$θ_{t + 1} = θ_{t} - η \cdot \frac{1}{B} \sum_{i = 1}^{B} \nabla_{θ} ℓ (θ_{t}, x_{i})$

Problem: Har gradient $\nabla ℓ (θ, x_{i})$ individual $x_{i}$ ke baare mein unbounded information contain kar sakta hai.

DP-SGD Fix (Abadi et al., 2016):

Har gradient clip karo: $\overset{g}{ˉ}_{i} = g_{i} / max (1, ∥ g_{i} ∥_{2} / C)$
- Kyun? Har per-example gradient norm ko $C$ tak bound karta hai. Add/remove neighboring definition ke under, clipped gradient ko sum $\sum_{i} \overset{g}{ˉ}_{i}$ se remove karne par woh at most $C$ change hota hai, isliye summed gradient ki sensitivity $Δ = C$ hai. (Replace-one definition ke under ye $2 C$ hoti, kyunki aap norm $\leq C$ wala ek contribution remove karte ho aur ek add karte ho.) Hum throughout add/remove use karte hain.
Gaussian noise add karo: Summed clipped gradient $\sum_{i} \overset{g}{ˉ}_{i}$ mein noise add karo jiska standard deviation sensitivity $C$ ke proportional ho: $\tilde{g} = \frac{1}{B} (\sum_{i = 1}^{B} \overset{g}{ˉ}_{i} + N (0, σ^{2} C^{2} I))$
- Bahar $1/ B$ kyun? Noise sum ki sensitivity ( $\sim σ C$ ) ke hisaab se calibrate hota hai, phir poora sum (signal + noise) $1/ B$ se average hota hai. Equivalently, averaged gradient mein add hone wale noise ka standard deviation per coordinate $σ C / B$ hai. Dono statements same mechanism describe karte hain; noise sum ki sensitivity $C$ ke liye scale hota hai averaging se pehle.
- Gaussian kyun? Hum (ε, δ)-DP (approximate DP) use karte hain jahan chhota failure probability δ acceptable hai.
Update: $θ_{t + 1} = θ_{t} - η \tilde{g}$

Privacy accounting: Poisson subsampling rate $q = B / N$ ke saath $T$ steps ke baad, privacy degrade hoti hai. Subsampled Gaussian mechanism ke liye moments accountant / advanced-composition bound hai:

$ε (T) \approx q 2 T ln (1/ δ) / σ$

Ye scaling kyun? Privacy loss $T$ ke roop mein accumulate hoti hai (linearly nahi!) advanced composition ki wajah se, aur subsampling se $q = B / N$ factor ke through amplify hoti hai: ek individual sirf $q$ fraction of steps mein appear karta hai. Dhyan do ki sensitivity $C$ aur batch size $B$ yahan alag-alag appear nahi karte — ye noise multiplier $σ$ mein absorb ho jaate hain (noise std ka sensitivity se ratio), jo actually privacy control karta hai.

Worked example Example 2: Binary Classifier ke liye DP-SGD

Setup: $N = 1000$ records pe DP ke saath logistic regression train karo (add/remove definition).

Parameters:

Batch size $B = 250$ → subsampling rate $q = B / N = 0.25$
Clipping norm $C = 1.0$
Noise multiplier $σ = 4.0$
Epochs $E = 10$ → $T = 400$ steps
Target $δ = 1 0^{- 5}$

Step 1: Per-example gradient clipping

for x_i in batch:
   g_i = compute_gradient(x_i)
   g_i = g_i / max(1.0, ||g_i|| / C)   # bound norm to C

Clip kyun? Unbounded gradients → unbounded sensitivity → unbounded privacy loss.

Step 2: Aggregate karo aur noise add karo (sum ki sensitivity C ke liye calibrate kiya)

g_sum   = sum(clipped_gradients)
noise   = Normal(0, sigma * C)          # std ~ sigma*C on the SUM
g_private = (g_sum + noise) / B         # then average
# equivalently, noise on the AVERAGE has std sigma*C/B

Sum pe $σ C$ kyun? Summed gradient ki sensitivity $C$ hai; noise multiplier $σ$ us sensitivity ke relative noise ko scale karta hai.

Step 3: Privacy accounting $q = 0.25$ , $T = 400$ , $σ = 4.0$ , $δ = 1 0^{- 5}$ ke saath subsampled-Gaussian bound use karke: $ε \approx q \frac{2 T l n ( 1/ δ )}{σ} = 0.25 \cdot \frac{2 \cdot 400 \cdot l n ( 1 0 ^{5} )}{4.0} \approx 0.25 \cdot \frac{9210}{4.0} \approx 0.25 \cdot \frac{95.97}{4.0} \approx 6.0$

Result: Is (loose) analytic bound ke under roughly ek $(6, 1 0^{- 5})$ -DP model. Practice mein ek numerically tight moments/RDP accountant same $σ$ ke liye chhota ε report karega; upar wala closed form ek upper-bound approximation hai. Takeaway: exact number heavily accountant pe depend karta hai — hamesha real accountant se ε compute karo, kabhi andaaze se nahi.

Membership Inference Attacks: The Threat

Models Membership Kyun Leak Karte Hain

Core insight: Models training data pe overfit ho jaate hain, train vs test examples pe alag behave karte hain.

Signal 1: Confidence

Training examples → high confidence predictions
Test examples → lower confidence

Kyun? Model ne training examples ko kai baar "dekha" hai, details memorize kar li hain.

Signal 2: Loss value

Training loss < Test loss (overfitting ki definition se)
Agar $ℓ (f, x)$ unusually low hai, toh $x$ likely training set mein hai

Classic Attack (Shokri et al., 2017)

Setup:

Target model $f_{target}$ jo $D_{target}$ pe trained hai
Attacker ke paas $f_{target} (x)$ ka access hai (black-box queries)
Attacker similar data pe shadow models train karta hai

Attack Steps:

Shadow models train karo: $f_{shadow}^{(1)}, \dots, f_{shadow}^{(k)}$ datasets $D_{1}, \dots, D_{k}$ pe
- Kyun? Target model ke behavior ko simulate karo taaki training data generate ho sake
Attack training set banao:
- Har shadow model ke liye, examples ko "in" ya "out" of training label karo
- Features: prediction vector $f_{shadow} (x) = [p_{1}, \dots, p_{C}]$
Binary classifier train karo: $A (f (x)) \to {0, 1}$
- Input: prediction confidence scores
- Output: membership prediction
Target pe apply karo: $A (f_{target} (x))$ use karke membership infer karo

Worked example Example 3: Simple Membership Inference

Scenario: Medical diagnosis (diabetes) ke liye binary classifier, 1000 training examples.

Step 1: Confidence threshold compute karo

Training set pe average confidence: 0.92
Held-out validation pe average confidence: 0.78
Threshold set karo: $τ = 0.85$ (midpoint)

Step 2: Attack rule

def is_member(model, x):
   confidence = max(model.predict_proba(x))
   return confidence > 0.85

Ye threshold kyun? False positives (non-members ko members claim karna) aur false negatives ke beech balance karta hai.

Step 3: Evaluate karo

500 known train examples pe test: 380 correctly identified (76% recall)
500 known test examples pe test: 120 incorrectly identified (24% false positive rate)
AUC = 0.76

Interpretation: Significant privacy leak. Ek attacker 3/4 training examples identify kar sakta hai.

The DP-Privacy Connection

Intuition DP Membership Inference ke Khilaf Kyun Defend Karta Hai

Differential privacy directly membership inference advantage ko bound karta hai. Attacker ka true-positive rate (TPR = member detect karna) aur false-positive rate (FPR = non-member flag karna) define karo. DP dono distributions (member-in vs member-out) ko relate karta hai taaki:

$TPR \leq e^{ε} \cdot FPR + δ$

aur symmetrically. Isse advantage (members vs non-members pe success probabilities ka difference) bound hota hai:

$Adv \leq \frac{e ^{ε} - 1}{e ^{ε} + 1} + δ \approx \frac{ε}{2} (chhote ε ke liye)$

Proof sketch:

$D_{1}$ mein $x$ include ho, $D_{2}$ mein exclude ho
$D_{1}$ se model: $f_{1} = M (D_{1})$ ; $D_{2}$ se model: $f_{2} = M (D_{2})$
DP: $Pr [A (f_{1}) = 1] \leq e^{ε} Pr [A (f_{2}) = 1] + δ$
Dono directions combine karke aur simplify karke upar wala advantage bound milta hai.

Ye kyun matter karta hai: ε worst-case gap control karta hai ki attacker kitni baar members ko flag karta hai vs non-members ko. Ye success probabilities ke difference pe bound hai, multiplicative factor over "random guessing" nahi. Chhote ε ke liye chance se excess advantage roughly $ε /2$ hai.

Privacy-Utility Tradeoff

Fundamental tension:

$Utility \propto \frac{1}{σ ^{2}} \propto \frac{1}{ε ^{2}}$

Low ε (strong privacy): Bada noise chahiye → poor model accuracy
High ε (weak privacy): Kam noise → better accuracy lekin zyada leakage

Practical values:

ε < 1: Strong privacy, significant utility loss
ε = 1-10: Moderate privacy, acceptable utility
ε > 10: Weak privacy, minimal utility impact

Free lunch kyun nahi? Information-theoretically, learning ke liye data se signal extract karna zaroori hai. Privacy ke liye individual contributions chhupana zaroori hai. Ye goals conflict karte hain.

Worked example Example 4: Privacy-Utility Tradeoff Analysis

Setup: Image classification (CIFAR-10), privacy levels compare karte hue.

Privacy Budget	Accuracy	Attack AUC	Interpretation
Non-private	92%	0.82	Severe leakage
ε = 8, δ = 10⁻⁵	88%	0.64	Moderate leak
ε = 2, δ = 10⁻⁵	81%	0.53	Strong privacy
ε = 0.5, δ = 10⁻⁵	68%	0.51	Near-perfect privacy

Step-by-step interpretation:

Non-private baseline: High accuracy, lekin attackers AUC 0.82 se training data identify kar sakte hain (0.5 random se bahut better).
ε = 8: Modest privacy, sirf 4% accuracy drop. Attack abhi bhi kuch effective hai (AUC 0.64).

Abhi bhi vulnerable kyun? ε = 8 relatively bada hai; probability ratio $e^{8} \approx 2981$ tak allow karta hai.

ε = 2: Strong privacy guarantee. 11% accuracy drop, lekin attack barely better than random (0.53 vs 0.5).

Effective kyun? ε = 2 matlab ratio bound $e^{2} \approx 7.4$ hai — attackers ke liye exploit karna mushkil.

ε = 0.5: Near-perfect privacy (attack essentially random guessing hai), lekin 24% accuracy drop.

Itna costly kyun? Bahut chhota ε, signal ke relative bahut bada noise require karta hai.

Takeaway: Zyada tar applications ke liye, ε ≈ 2-4 sweet spot hai: strong privacy with manageable utility loss.

Common Mistakes

Common mistake Mistake 1: "Random noise add karna = differential privacy"

Ye sahi kyun lagta hai: Noise individual contributions ko mask karta hai.

Ye galat kyun hai: DP ke liye sensitivity ke based pe calibrated noise chahiye. Sensitivity analysis ke bina random noise koi formal guarantee provide nahi karta.

Example:

# WRONG
noisy_mean = true_mean + random.gauss(0, 0.1)
# RIGHT
sensitivity = max_value / n
noisy_mean = true_mean + laplace(0, sensitivity / epsilon)

Fix: Hamesha pehle sensitivity compute karo, phir accordingly noise scale karo.