Foundations — Federated learning

Before you can read the parent note, you need to own every symbol it throws at you. This page builds each one from absolute zero — plain words first, then a picture, then why the topic can't live without it. Read top to bottom: each idea is a brick for the next.

1. What is a "model" and what is $\theta$?

Start with the thing everything else acts on.

A model is a machine that takes an input (a photo, a half-typed sentence) and produces a guess (a label, the next word). Inside that machine are knobs — numbers you can turn to change its behaviour. Turn them one way and the model predicts "cat"; turn them another way and it predicts "dog".

Look at the figure. On the left is the abstract "machine" with dials; on the right, we've unrolled every dial into a single vertical column of numbers — that column is $\theta$. When the parent writes ${\theta }^{(0)}$, ${\theta }^{(t)}$, ${\theta }_k^{(t+1)}$, every one of those is just a different column of numbers — a different setting of the same dials.

• The little superscript in parentheses, ${\theta }^{(t)}$, means "the value at round number $t$" — a snapshot in time.
• The subscript ${\theta }_k$ means "the version living on client number $k$".

2. Clients, data, and the counting symbols $N$, $k$, $n_k$, $n$

Now we need to talk about many devices, each with some data.

The picture of the totals:

In the figure, each barrel is one client's data bag; the height of the barrel is $n_k$. The total volume poured into the big tank on the right is $n$. This picture is the whole reason for the weighting you'll meet next: a fat barrel should have more say than a thin one.

3. Loss $\ell$, and why we average it into $F_k$ and $F$

A model needs a way to know it is wrong. That is the job of loss.

Here $x_i$ is one input (the $i$-th photo), $y_i$ is its true answer, and $i$ is just another index like $k$ but pointing at one example inside a bag.

One client's total badness is the average loss over its whole bag:

Read it slowly: sum the badness over every example $i$ in the bag ${\mathcal{D}}_k$, then divide by how many there were ($n_k$) to get the average. We average (rather than just sum) so a big bag isn't automatically "worse" just for being big.

The global badness stitches all clients together, giving each a vote proportional to its size:

4. Minimising: $\arg \min$, gradient $\nabla$, and the step size $\eta$

We have a badness score $F(\theta )$. Training means hunting for the $\theta$ that makes it smallest.

To search, we need to know which way is downhill. That is the gradient.

The figure shows $F$ as a bowl-shaped valley (height = badness). A ball sits on the slope; the orange arrow is $\nabla F$ pointing uphill, the teal arrow $-\nabla F$ points into the valley. Rolling repeatedly along $-\nabla F$ lands us at the bottom ${\theta }^{\ast }$.

SGD ("stochastic gradient descent") is the same idea but estimating $\nabla F$ from a handful of examples at a time instead of the whole bag — faster, a little noisy. "Epoch" = one full pass through a client's bag; $E$ epochs = $E$ passes.

The gradient of the global loss is the weighted sum of client gradients

Before we can average anything, we need the single identity that makes the whole scheme legal. Recall $F(\theta )={\sum }_k\frac{n_k}{n}{F}_k(\theta )$. The gradient of a weighted sum is the weighted sum of the gradients (differentiation passes through addition and through constant weights untouched), so:

5. Rounds, client subsets ${\mathcal{S}}_t$, and the FedAvg update rule

Federated training happens in rounds. In each round the server doesn't bother every device (that's slow and drains batteries) — it picks a small sample.

Here $j$ is yet another index letter — same meaning as $k$, used to avoid confusion when two sums appear together.

What each client does in a round

A chosen client $k$ receives the current shared model ${\theta }^{(t)}$ and rolls it downhill on its own loss $F_k$. Writing the round number explicitly, one local gradient step is:

Read it: "start from the shared model ${\theta }^{(t)}$ that just arrived, take a downhill step on my loss, and call the result my new model ${\theta }_k^{(t+1)}$." (Doing $E$ epochs just repeats this step $E$ times before sending.) Notice the two index habits meeting at last: the superscript $(t+1)$ says which round, the subscript $k$ says which client.

How the server combines them — the FedAvg rule

The server now has one improved model ${\theta }_k^{(t+1)}$ from each chosen client. It forms the new shared model as the size-weighted average over the present clients:

Why this equals one centralised step

Put the two boxes together for the simplest case — every client takes one step and everyone participates (${\mathcal{S}}_t$ = all $N$ clients, so $n_{{\mathcal{S}}_t}=n$). Substitute the local update into the aggregation:

The last equality is exactly the identity from Section 4. So one round of FedAvg with one local step is one honest gradient-descent step on the global loss $F$ — the server reproduces centralised training without ever touching the data.

6. Symbols for privacy: $\mathcal{N}(0,{\sigma }^2)$, $S$, $(ϵ,\delta )$

The later part of the parent guards against leakage. Meet its alphabet.

The prerequisite map

Read it as a flow: knobs and loss make a per-client score; counts turn many scores into one weighted global score; gradients let each client descend; rounds and subsets glue their descents together into FedAvg; privacy noise wraps the whole thing. All arrows drain into the topic, Federated learning.

Adjacent tools you can safely postpone until later child pages: Optimization Theory (the convergence proofs), Edge Computing (where clients physically live), Model Compression (shrinking what $\theta$ costs to send), Byzantine Robust Learning and Multi-Task Learning (handling malicious or personalised clients).

Equipment checklist

Cover the right side and answer aloud. If any answer surprises you, re-read its section above.