Visual walkthrough — Federated learning

We only assume you can add fractions and picture an arrow pointing "downhill". Everything else — loss, gradient, weighted average — we build from scratch.

---

Step 1 — What is a "model", and what is "loss"?

WHAT. A model is just a bag of numbers we can tune. Call that bag $\theta$ (Greek letter "theta"). Think of $\theta$ as the position of a hiker on a landscape.

WHY. Learning = finding the lowest point of a landscape called the loss. The loss $F(\theta )$ is one number: "how wrong is the model $\theta$ right now?" Low loss = good model. So learning is literally "walk downhill until you can't go lower".

PICTURE. The curve below is the loss landscape. The horizontal axis is $\theta$ (every possible model). The vertical axis is $F(\theta )$ (how wrong). The red dot is our current model; the star is the best model ${\theta }^{\ast }$ we want to reach.

---

Step 2 — The gradient: which way is downhill?

WHAT. At the hiker's position, the ground has a slope. The gradient $\nabla F(\theta )$ ("nabla F") is an arrow pointing in the direction the ground goes up the fastest.

WHY this tool and not another? We need a rule that says which way to step. A plain slope (a derivative) only works in one dimension; real models have millions of numbers. The gradient is the honest multi-dimensional generalisation of "slope" — it is the unique arrow of steepest ascent. To go down, we step opposite to it, i.e. towards $-\nabla F$.

PICTURE. The orange arrow is the gradient (points uphill). We move against it. The step size $\eta$ ("eta") controls how far we walk. One downhill step:

$$\underset{\text{where we land}}{\underbrace{{\theta }_{\text{new}}}}=\underset{\text{where we are}}{\underbrace{\theta }}-\underset{\begin{array}{c}\text{step}\\ \text{size}\end{array}}{\underbrace{\eta }}\underset{\begin{array}{c}\text{uphill}\\ \text{arrow}\end{array}}{\underbrace{\nabla F(\theta )}}$$

The minus sign is the whole point: it flips "uphill" into "downhill".

---

Step 3 — The data lives in scattered pieces

WHAT. In the real world the training data is not in one pile. It sits on $N$ separate devices ("clients"). Client number $k$ holds a private stash ${\mathcal{D}}_k$ with $n_k$ examples in it.

WHY. We are not allowed to gather the piles into one — that is the whole privacy premise of federated learning (Privacy-Preserving ML, Edge Computing). So each client can only measure the loss on its own pile. Client $k$'s private loss is $F_k(\theta )$: "how wrong is $\theta$, judged only by client $k$'s data".

PICTURE. Three phones, three separate data piles, three separate little loss curves — one per client. Nobody sees the others' data.

---

Step 4 — The global loss is a weighted blend

WHAT. The loss we truly care about is over everyone's data at once. Call the grand total $n=n_1+n_2+\cdots +n_N$. The global loss is a weighted average of the per-client losses:

$$F(\theta )=\sum _{k=1}^N\underset{\begin{array}{c}\text{client }k\text{’s}\\ \text{share of data}\end{array}}{\underbrace{\frac{n_k}{n}}}\underset{\begin{array}{c}\text{client }k\text{’s}\\ \text{own loss}\end{array}}{\underbrace{F_k(\theta )}}$$

WHY the weights $n_k/n$? Each example should count once. A client with $10,000$ examples deserves $100\times$ more say than one with $100$. The fraction $n_k/n$ is exactly "what slice of all the data belongs to client $k$". These fractions are positive and add up to $1$ — that is what makes it an honest average and not a random sum.

PICTURE. A pie chart: the whole pie is $n$ (all data). Each wedge is one client's slice $n_k/n$. The global loss is these wedges scaled by each client's own loss and stacked up.

---

Step 5 — Give every client the SAME starting model

WHAT. At the start of a round $t$, the server holds one global model ${\theta }^{(t)}$ and broadcasts it to the selected clients. Every client begins its local work from the identical point ${\theta }^{(t)}$.

WHY this matters for the proof. The magic in the next step only works because everyone starts at the same ${\theta }^{(t)}$. That shared starting point is the hinge the whole derivation swings on — keep your eye on it.

PICTURE. One model ${\theta }^{(t)}$ leaves the server and lands, unchanged, as the identical red dot on each client's private loss curve.

---

Step 6 — Each client takes one local downhill step

WHAT. Client $k$ runs gradient descent on its own loss $F_k$, starting from the shared ${\theta }^{(t)}$. For the clean proof we let each client take exactly one step:

$${\theta }_k^{(t+1)}=\underset{\begin{array}{c}\text{shared}\\ \text{start}\end{array}}{\underbrace{{\theta }^{(t)}}}-\eta \underset{\begin{array}{c}\text{client }k\text{’s own}\\ \text{downhill arrow}\end{array}}{\underbrace{\nabla F_k({\theta }^{(t)})}}$$

WHY. Each client can only see its own data, so it can only compute its own gradient $\nabla F_k$. Different clients point in different directions — client A's downhill is not client B's downhill. They walk off toward their own private minima.

PICTURE. Three red dots that started stacked now split apart, each sliding down its own curve to a different landing spot ${\theta }_k^{(t+1)}$.

---

Step 7 — Averaging the models = one global gradient step

This is the punchline. WHAT the server does: it takes the same weighted average from Step 4, but applied to the models the clients sent back:

$${\theta }^{(t+1)}=\sum _{k=1}^N\frac{n_k}{n}{\theta }_k^{(t+1)}$$

WHY it works — watch every term. Substitute what each client actually sent (Step 6), then split the sum:

$${\theta }^{(t+1)}=\sum _k\frac{n_k}{n}({\theta }^{(t)}-\eta \nabla F_k({\theta }^{(t)}))$$ $$=\underset{=1}{\underbrace{\left(\sum _k\frac{n_k}{n}\right)}}{\theta }^{(t)}-\eta \underset{=\nabla F({\theta }^{(t)})}{\underbrace{\sum _k\frac{n_k}{n}\nabla F_k({\theta }^{(t)})}}$$

Two facts make it collapse:

1. The weights add up to 1 (${\sum }_k{n}_k/n=n/n=1$), so the ${\theta }^{(t)}$ terms merge back into a single ${\theta }^{(t)}$.
2. The weighted sum of the per-client gradients is the global gradient, because $F={\sum }_k\frac{n_k}{n}{F}_k$ and the gradient of a weighted sum is the weighted sum of gradients (differentiation is linear).

So the whole thing becomes:

$$\boxed{{\theta }^{(t+1)}={\theta }^{(t)}-\eta \nabla F({\theta }^{(t)})}$$

That is exactly one gradient-descent step on the global loss $F$ — the rule from Step 2 — even though nobody ever pooled the data.

PICTURE. The three scattered landing dots get pulled back to their weighted average (the blue dot). That blue dot lands precisely where a single global downhill step from ${\theta }^{(t)}$ would have landed (dashed arrow).

---

Step 8 — The edge case: what breaks when $E>1$?

WHAT. The clean equality of Step 7 needed each client to take one step. In practice we let each client take $E$ steps ("local epochs") to save communication. Now client $k$ walks $E$ times down its own curve before reporting.

WHY it stops being exact. After the first step client $k$ is no longer at ${\theta }^{(t)}$, so its second gradient is measured somewhere else — a place no other client visited. The clients drift toward their own private minima. When their data is un-alike (non-IID — see Multi-Task Learning), those minima disagree, and the average lands off the true global step.

PICTURE. With $E=1$ the average sits on the true path. With $E=5$ the drifted average pulls sideways — an error that grows with $E$ and with how mismatched the clients are.

The convergence bound from the parent note names both effects:

$$F({\theta }^{(T)})-F({\theta }^{\ast })\le \underset{\begin{array}{c}\text{shrinks with}\\ \text{more rounds }T\end{array}}{\underbrace{\mathcal{O}\left(\frac{1}{T}\right)}}+\underset{\begin{array}{c}\text{drift floor}\\ \text{grows with }E\end{array}}{\underbrace{\mathcal{O}(E^2{\eta }^2{\sigma }^2)}}$$

Everything on the noise/robustness side (clipping, Differential Privacy, Byzantine Robust Learning, Model Compression of the updates) sits on top of this skeleton — it protects and shrinks the messages, but the skeleton is Steps 5→6→7.

---

The one-picture summary

One round of FedAvg, front to back: broadcast the same model → each client steps down its own private curve → weighted-average the returned models → land exactly where one global gradient step would have landed.

Recall Feynman retelling — say it in plain words

The server has one hiker and copies it onto every phone at the same spot. Each phone lets its copy take a small step downhill on its own private hill — and since every phone starts at the same place, "how far each copy moved" is just "the slope of that phone's hill". The server then places a new hiker at the weighted average of where all the copies ended up, giving big-data phones a bigger pull. Because slopes add up the same way the losses do, that average spot is identical to the spot one hiker would reach if it could see everybody's data at once. No data ever left a phone. The only crack: if each copy takes many steps before reporting, it wanders off toward its own phone's private valley, and the average drifts a little — worse when phones hold very different data. That drift is the price we pay for talking to the server less often.

---

Prerequisites & neighbours: Optimization Theory · Edge Computing · Differential Privacy · Privacy-Preserving ML · Byzantine Robust Learning · Model Compression · Multi-Task Learning · back to Federated learning