Worked examples — Federated learning

This page is a workout. The parent note built the machinery — the loss $F(\theta )$, the FedAvg averaging rule, non-IID data, and differential privacy. Here we run that machinery through every case it can face and check each answer by hand.

Before anything, let us make sure two words never sneak up on you:

---

The scenario matrix

Federated learning problems fall into a handful of case classes. Every worked example below is tagged with the cell it lands in, and together they cover the whole grid.

Cell	Case class	The twist it tests
A	Balanced clients	Equal $n_k$ — does weighted = plain average?
B	Unbalanced clients	Very different $n_k$ — who pulls the model?
C	Degenerate input	A client with $n_k=0$ (dropout / empty device)
D	Multi-epoch drift	$E>1$ local epochs — the gap vs pure gradient descent
E	Communication cost	Bytes on the wire, federated vs centralized
F	Non-IID conflict	Clients whose optima ${\theta }_k^{\ast }$ point opposite ways
G	Privacy noise	Adding DP noise, and what it costs in accuracy
H	Exam twist	A "spot the trap" question combining B + C

---

Case A — Balanced clients

1. Write the totals. $n=100+100+100=300$. Why this step? The FedAvg rule ${\theta }^{(t+1)}={\sum }_k\frac{n_k}{n}{\theta }_k$ needs the denominator $n$ first — it is the shared "size of the pie."
2. Compute each weight fraction. $\frac{n_k}{n}=\frac{100}{300}=\frac{1}{3}$ for all three. Why this step? When counts are equal, every fraction collapses to $1/N$ — the weighted average becomes the plain average. This is the sanity anchor for all later cases.
3. Average. $\theta =\frac{1}{3}(2.0)+\frac{1}{3}(4.0)+\frac{1}{3}(6.0)=4.0$. Why this step? Blending the models is literally a coordinate-wise weighted mean.

Verify: The plain mean of $2,4,6$ is $12/3=4.0$. Weighted and plain agree exactly — as they must when $n_k$ are equal. ✓ (Units: weights are unitless model parameters.)

---

Case B — Unbalanced clients

1. Totals. $n=900+100=1000$.
2. Fractions. $\frac{n_A}{n}=0.9$, $\frac{n_B}{n}=0.1$. Why this step? This $0.9$ vs $0.1$ split is the whole point of the [!mistake] in the parent note — data size is voting power.
3. Weighted average. $\theta =0.9(1.0)+0.1(11.0)=0.9+1.1=2.0$.
4. Naive average (the trap). $\frac{1}{2}(1.0+11.0)=6.0$.

Look at the figure: the black tick (weighted $=2.0$) sits close to the big client A, while the grey tick (plain $=6.0$) drifts toward the tiny, noisy client B.

Verify: Sanity check the extremes: if B had $n_B=0$ we'd expect $\theta ={\theta }_A=1.0$. Our formula gives $0.9\to 1.0$ as $n_B\to 0$. The weighted answer $2.0$ sits between, closer to A — correct. ✓

---

Case C — Degenerate input (empty client)

1. Totals over the sampled set. $n_{\mathcal{S}}=200+300+0=500$. Why this step? FedAvg's denominator is ${\sum }_{j\in \mathcal{S}}{n}_j$ — the total of this round's clients, not all clients.
2. Fractions. $\frac{200}{500}=0.4,\frac{300}{500}=0.6,\frac{0}{500}=0$. Why this step? The empty client gets weight exactly zero — the maths deletes it automatically. This is the degenerate case working by design.
3. Average. $\theta =0.4(5.0)+0.6(10.0)+0\cdot (999.0)=2.0+6.0+0=8.0$.

Verify: Drop the empty client entirely and re-run: $\frac{200}{500}(5)+\frac{300}{500}(10)=8.0$. Identical — proving the $n_k=0$ term is genuinely inert. ✓

---

Case D — Multi-epoch drift ($E>1$)

1. One true GD step. $\theta ={\theta }^{(0)}-\eta \nabla F({\theta }^{(0)})=1.0-0.1(1.0)=0.9$. Why this step? The parent note proved: with $E=1$, FedAvg $\equiv$ one global gradient step. This is our baseline.
2. Local epoch 1. $\theta =1.0-0.1(1.0)=0.9$.
3. Local epoch 2. Re-evaluate the gradient at the new point: $\theta =0.9-0.1(0.9)=0.81$. Why this step? Multi-epoch training keeps stepping on the client's own loss, not the global one — this recomputation is exactly where drift is born.
4. Local epoch 3. $\theta =0.81-0.1(0.81)=0.729$. Why this step? Notice the pattern $\theta =(1-\eta {)}^E=0.9^3$. Three epochs travelled to $0.729$, well past the single-step $0.9$.

The orange curve races down faster with $E=3$: fewer communication rounds, but on a many-client problem each client races toward its own minimum — that is the $\mathcal{O}(E^2{\eta }^2{\sigma }^2)$ drift term. See Optimization Theory for the convergence machinery.

Verify: Closed form $(1-\eta {)}^E=0.9^3=0.729$. Matches step 4 exactly. ✓ For a single client (no heterogeneity, $\sigma =0$) more epochs is pure speedup — drift only bites across differing clients.

---

Case E — Communication cost

1. Bytes per model transfer. $P\times 4=5\text{M}\times 4=20\text{MB}$. Why this step? We ship weights, not data — the model size is fixed regardless of dataset size.
2. Per round, both directions, all clients. $50\times 2\times 20\text{MB}=2000\text{MB}=2\text{GB}$.
3. All rounds. $20\times 2\text{GB}=40\text{GB}$.
4. Centralized upload. $2,000,000\times 120\text{KB}=2.4\times 10^8\text{KB}=240\text{GB}$. Why this step? Centralized cost scales with data volume; federated with model size × rounds. That difference is the whole economic argument, tied to Edge Computing.
5. Ratio. $240/40=6\times$ cheaper here.

Verify: Units line up: $\text{images}\times \frac{\text{KB}}{\text{image}}=\text{KB}$; $240\text{GB}=240,000\text{MB}$; $240,000/40,000=6$. ✓

---

Case F — Non-IID conflict

1. Model the global loss. With equal weight and quadratic local losses centred at each optimum, $F(\theta )=\frac{1}{2}[(\theta +3{)}^2+(\theta -9{)}^2]$. Why this step? A quadratic bowl per client is the standard local model near an optimum; averaging bowls gives the global bowl.
2. Minimise. $F^{\prime }(\theta )=(\theta +3)+(\theta -9)=2\theta -6=0\Rightarrow {\theta }^{\ast }=3$. Why this step? With equal weights the global optimum is the midpoint of the two client optima — the compromise no client is happy with.
3. Divergences. ${\delta }_1=(-3-3{)}^2=36$, ${\delta }_2=(9-3{)}^2=36$. Why this step? Large, symmetric ${\delta }_k$ is the signature of harmful non-IID data — both clients get dragged $6$ units from where they wanted to be. This is the case Multi-Task Learning and personalisation aim to soften.

Verify: Midpoint of $-3$ and $9$ is $(-3+9)/2=3$. ✓ And $F^{\prime \prime }(\theta )=2>0$ confirms it's a minimum, not a maximum. ✓

---

Case G — Privacy noise

1. Per-client noise std. $\sigma S=0.05\times 1.0=0.05$. Why this step? Clipping to $S$ first bounds sensitivity — without it we'd need infinite noise (parent note's Gboard example). See Differential Privacy.
2. Variance adds when averaging independent noises. Averaging $M$ independent draws each of variance $v$ gives variance $v/M$. Why this step? Independent randomness cancels — this is why federating over many clients helps privacy-vs-accuracy: noise averages out but the signal doesn't.
3. Aggregated std. $\frac{\sigma S}{\sqrt{M}}=\frac{0.05}{\sqrt{50}}\approx 0.00707$.
4. Accuracy impact. The reported weight is $4.00\pm 0.0071$ — a $\approx 0.18\%$ perturbation. Tolerable.

Verify: $\sqrt{50}\approx 7.071$; $0.05/7.071=0.007071$; $0.007071/4.00=0.00177=0.18\%$. ✓ Trade-off holds: raise $\sigma$ to $0.5$ and the perturbation jumps to $1.8\%$ — stronger privacy, weaker accuracy, exactly as in Privacy-Preserving ML.

---

Case H — Exam twist (combines B + C + a robustness trap)

1. Naive plain average (trap 1). $\frac{1}{4}(2+3+50+1000)=263.75$ — dominated by both the empty client's junk and the attacker. Rejected.
2. Standard FedAvg with reported sizes. $n_{\mathcal{S}}=400+600+0+1000=2000$. Fractions $0.2,0.3,0,0.5$. Weighted: $0.2(2)+0.3(3)+0+0.5(1000)=0.4+0.9+500=501.3$. Why this step? Weighting did kill the empty client (Case C) — but the attacker declared the largest $n_k$, so weighting amplifies it. FedAvg alone is not robust.
3. The lesson. Size-weighting defends against noise-from-small-clients but not against a lying large client. You need a robust aggregator (coordinate median / trimmed mean) — see Byzantine Robust Learning.
4. Trimmed-mean sketch. Drop the top and bottom value, average the middle: honest weights near $2$–$3$ survive, the $1000$ outlier and the $50$ junk are discarded, giving $\approx 2.5$.

Verify: Naive: $(2+3+50+1000)/4=1055/4=263.75$. ✓ FedAvg: $0.4+0.9+0+500=501.3$. ✓ Both "correct by formula" yet both wrong for safety — that is the exam trap: the arithmetic is fine, the threat model is not.

---

Recall Quick self-test

Equal $n_k$ makes weighted average equal to ::: the plain average. A client with $n_k=0$ receives FedAvg weight ::: exactly zero (it drops out). After $E$ local epochs on $F=\frac{1}{2}{\theta }^2$ with $\eta =0.1$, one weight becomes ::: $(1-\eta {)}^E=0.9^E$. The global optimum of two equal, opposing clients sits at ::: the midpoint of their optima. Averaging $M$ independent DP noise draws shrinks the noise std by a factor of ::: $\sqrt{M}$. Size-weighting protects against small noisy clients but NOT against ::: a Byzantine client lying about a huge $n_k$.