Federated learning

6.5.8 · AI-ML › Research Frontiers & Practice

Yeh Kaunsi Problem Solve Karta Hai?

Classic centralized ML workflow:

1. Millions of devices se user data collect karo → central server
2. Aggregated dataset pe model train karo
3. Model wapas devices pe deploy karo

Problems:

• Privacy: Sensitive data (health records, messages, photos) user ke control se bahar chali jaati hai
• Bandwidth: Terabytes of raw data transmission
• Regulatory: GDPR, HIPAA certain data ko move karne se rokti hain
• Latency: Har training batch ke liye cloud ka round-trip

Federated learning data ko decentralized rakhta hai phir bhi ek global model seekhta hai.

Federated Learning Kaise Kaam Karta Hai (Step-by-Step Derivation)

Setup

Hamare paas hai:

• $N$ clients (devices, hospitals, organizations), har ek ke paas local dataset ${\mathcal{D}}_k$ jiska size $n_k$ hai
• Total data points: $n={\sum }_{k=1}^N{n}_k$
• Goal: Ek shared model $\theta$ seekhna jo global loss minimize kare:

jahan $F_k(\theta )=\frac{1}{n_k}{\sum }_{i\in {\mathcal{D}}_k}\ell (x_i,y_i;\theta )$ client $k$ ki local loss hai.

Yeh weighting kyun? Har client apne data size ke proportional contribute karta hai. 1000 samples wale client ka model pe 10× zyada influence hona chahiye 100 samples wale se.

Algorithm: Federated Averaging (FedAvg)

Yeh breakthrough McMahan et al. (2017) algorithm hai:

Server (orchestrator) steps:

1. Initialize global model ${\theta }^{(0)}$
2. Har round $t=0,1,2,\dots$ ke liye:
   • $C\cdot N$ clients ka subset ${\mathcal{S}}_t$ sample karo (typically $C=0.1$, yaani 10%)
   • Selected clients ko current ${\theta }^{(t)}$ broadcast karo
   • Client updates ka wait karo
   • Aggregate: ${\theta }^{(t+1)}={\sum }_{k\in {\mathcal{S}}_t}\frac{n_k}{n_{{\mathcal{S}}_t}}{\theta }_k^{(t+1)}$ jahan $n_{{\mathcal{S}}_t}={\sum }_{k\in {\mathcal{S}}_t}{n}_k$

Client $k$ steps (jab select hota hai):

1. Receive ${\theta }^{(t)}$ server se
2. Local training: Local data ${\mathcal{D}}_k$ pe $E$ epochs of SGD run karo: ${\theta }_k^{(t+1)}={\theta }^{(t)}-\eta {\sum }_{e=1}^E\nabla F_k(\theta )$
3. Send back ${\theta }_k^{(t+1)}$ (sirf updated weights, data nahi!)

Yeh kyun kaam karta hai—math:

Agar har client ek SGD step kare ($E=1$, batch size $n_k$): ${\theta }_k^{(t+1)}={\theta }^{(t)}-\eta \nabla F_k({\theta }^{(t)})$

Server aggregation: ${\theta }^{(t+1)}={\sum }_k\frac{n_k}{n}\left({\theta }^{(t)}-\eta \nabla F_k({\theta }^{(t)})\right)={\theta }^{(t)}-\eta {\sum }_k\frac{n_k}{n}\nabla F_k({\theta }^{(t)})$

Yeh exactly $F(\theta )$ pe gradient descent ka ek step hai! Multiple local epochs ($E>1$) isko zyada efficiently approximate karte hain.

Key Concepts Deep Dive

1. Communication Efficiency

Bottleneck: Networks pe model updates bhejna expensive hai (mobile data, slow hospital networks).

Kitne communication rounds chahiye accuracy ke liye?

Convergence analysis se derived (simplified):

Strongly convex $F$ ke liye, $T$ rounds ke baad: $F({\theta }^{(T)})-F({\theta }^{\ast })\le \mathcal{O}\left(\frac{1}{T}\right)+\mathcal{O}(E^2{\eta }^2{\sigma }^2)$

• Pehla term: Zyada rounds $T$ se faster convergence
• Doosra term: Local drift se error (clients $E$ local epochs karne pe diverge karte hain)

Trade-off:

• Bada $E$ → kam rounds (kam communication) lekin zyada drift error
• Chhota $E$ → zyada rounds (zyada communication) lekin kam drift

Practical choice: $E=5$ se $E=20$ epochs per round zyada tar tasks ke liye accha kaam karta hai.

2. Non-IID Data Distribution

Real-world data clients ke across heterogeneous hoti hai:

• Label skew: User A sirf English mein type karta hai, User B Spanish mein
• Feature skew: Hospital A ke older patients hain, Hospital B ke younger
• Quantity skew: Kuch users 10× zyada data generate karte hain

Mathematical impact:

Local aur global optima ke beech divergence define karo: ${\delta }_k=\Vert {\theta }_k^{\ast }-{\theta }^{\ast }{\Vert }^2$

jahan ${\theta }_k^{\ast }=\arg \min F_k(\theta )$ (client $k$ ka optimal) aur ${\theta }^{\ast }=\arg \min F(\theta )$ (global optimal) hai.

High ${\delta }_k$ → clients conflicting directions mein kheenchte hain → slower convergence.

Mitigation strategies:

1. Zyada local epochs aggregating se pehle har client ko better local minimum dhundhne mein help karta hai
2. Client sampling: Har round mein diverse clients sample karo
3. Personalization: Clients ko small personalized layers rakhne do

3. Privacy Guarantees

Claim: "Federated learning private hai kyunki raw data kabhi devices se bahar nahi jaata."

Reality: Isse zyada nuanced hai.

Attack vectors:

1. Gradient inversion: Gradients se training samples reconstruct karo

   • ${\nabla }_{\theta }\ell (x,y;\theta )$ diya, $x$ ke liye solve karo
   • Chhote batches, high-res images ke liye possible hai

2. Model memorization: Neural nets training examples yaad rakh lete hain

   • Model ko prompts se query karo → woh training data regurgitate karta hai

Defense: Differential Privacy

Bhejna se pehle updates mein calibrated noise add karo:

${\tilde{\theta }}_k^{(t+1)}={\theta }_k^{(t+1)}+\mathcal{N}(0,{\sigma }^2{S}^{2}I)$

jahan $S$ clipping threshold hai (gradient norm bound karo).

Trade-off: Zyada noise ($\sigma \uparrow$) → stronger privacy → lower accuracy.

Advanced Variants

Secure Aggregation

Problem: Agar server malicious hai toh model updates bhi info leak kar sakte hain.

Solution: Cryptographic protocol jahan server ${\sum }_k{\theta }_k$ seekhta hai lekin individual ${\theta }_k$ nahi.

Kaise (simplified):

1. Har client $k$ ek random mask $r_k$ generate karta hai jo doosre clients ke saath share hota hai (pairwise keys ke zariye)
2. Client $k$ server ko ${\theta }_k+r_k$ bhejta hai
3. Server sum karta hai: ${\sum }_k({\theta }_k+r_k)={\sum }_k{\theta }_k+\underset{=0\text{ if designed right}}{\underbrace{\sum _k{r}_k}}$
4. Masks cancel ho jaate hain, server individuals ko dekhe bina true aggregate paata hai

Cost: 2-3× zyada communication, cryptographic overhead.

Cross-Silo vs. Cross-Device

Yeh distinction kyun matter karta hai: Algorithm hyperparameters ($C$, $E$, learning rate) ko setting ke hisaab se adapt karna padta hai.

Worked Example: Binary Classification

Setup: 3 hospitals bina records share kiye patient vitals se disease predict karna chahte hain.

Data:

• Hospital 1: 1000 patients, 60% positive
• Hospital 2: 500 patients, 40% positive
• Hospital 3: 2000 patients, 70% positive
• Feature: Blood pressure (simplicity ke liye 1D)
• Model: Logistic regression $p(y=1|x)=\sigma ({\theta }^{T}x)$

Loss: $F_k(\theta )=\frac{1}{n_k}{\sum }_{i=1}^{n_k}-[y_i\log \sigma ({\theta }^T{x}_i)+(1-y_i)\log (1-\sigma ({\theta }^T{x}_i))]$

Algorithm execution:

Round 0:

• Server: ${\theta }^{(0)}=0$ initialize karo, saare 3 hospitals sample karo ($C=1$)

• Hospital 1: ${\theta }^{(0)}=0$ receive karta hai

  • 1000 samples pe 5 epochs SGD run karta hai, $\eta =0.01$
  • Gradients compute karta hai: $\nabla F_1(\theta )=\frac{1}{1000}{\sum }_i(p_i-y_i)x_i$ jahan $p_i=\sigma ({\theta }^T{x}_i)$
  • Update karta hai: ${\theta }_1^{(1)}={\theta }^{(0)}-\eta \nabla F_1=0.42$ (maano)
  • Yeh step kyun? Standard gradient descent—local data pe loss reduce karne ke liye weights move karo.
  • ${\theta }_1^{(1)}=0.42$ wapas bhejta hai

• Hospital 2: Similarly ${\theta }_2^{(1)}=0.38$ paata hai

• Hospital 3: Similarly ${\theta }_3^{(1)}=0.51$ paata hai

• Server aggregate karta hai: ${\theta }^{(1)}=\frac{1000}{3500}\cdot 0.42+\frac{500}{3500}\cdot 0.38+\frac{2000}{3500}\cdot 0.51$ $=0.286\cdot 0.42+0.143\cdot 0.38+0.571\cdot 0.51=0.465$

  Yeh weights kyun? Total $n=1000+500+2000=3500$. Hospital 3 ke paas sabse zyada data hai (57%), isliye woh ${\theta }^{(1)}$ ko apne ${\theta }_3^{(1)}=0.51$ ke sabse kareeb kheenchta hai.

Round 1: ${\theta }^{(1)}=0.465$ se repeat karo ...

100 rounds ke baad: ${\theta }^{(100)}\approx 0.52$ (converged global model)

Privacy preserved: Hospitals ne kabhi patient records share nahi kiye, sirf learned model weights.

Common Pitfalls & How to Fix Them

Connections to Other Concepts

• Distributed Training: Federated learning privacy constraints aur non-IID data ke saath distributed training hai
• Differential Privacy: Federated updates ke liye formal privacy guarantee
• Optimization Theory: FedAvg SGD variant hai; convergence data heterogeneity pe depend karta hai
• Edge Computing: Federated learning computation ko edge devices pe push karta hai
• Multi-Task Learning: Personalized federated learning har client ko ek task treat karta hai
• Byzantine Robust Learning: Malicious clients ke khilaf defense jo bad updates bhejte hain
• Model Compression: Quantization + sparsity har round mein communication cost reduce karte hain
• Privacy-Preserving ML: Federated learning ek pillar hai; doosron mein homomorphic encryption, secure multi-party computation shamil hain

Active Recall Questions

#flashcards/ai-ml

Federated learning aur centralized ML mein core difference kya hai? :: Federated learning MODEL ko DATA ke paas laata hai (training local devices pe hoti hai, sirf weight updates bheje jaate hain). Centralized ML DATA ko MODEL ke paas laata hai (raw data training ke liye central server pe upload hota hai).

Federated averaging update rule likho :: ${\theta }^{(t+1)}={\sum }_{k\in {\mathcal{S}}_t}\frac{n_k}{{\sum }_{j\in {\mathcal{S}}_t}{n}_j}{\theta }_k^{(t+1)}$ jahan ${\theta }_k^{(t+1)}$ client $k$ ka local model hai $E$ epochs of SGD ke baad, dataset size $n_k$ se weighted.

Concept Map