Kubernetes for ML workloads: Visual walkthrough

This is a deep-dive child of Kubernetes for ML workloads. It leans on the control-loop idea from Control Systems - Feedback Loops and the scaling picture from Horizontal vs Vertical Scaling.

Step 1 — What is a "replica", and what is a "metric"?

WHAT. A replica is one running copy of your model server — an identical worker. If you have 4 replicas, four identical boxes are all answering requests. A metric is one number we measure per replica that tells us how hard it is working — for example its CPU busy-ness as a percentage, or how many requests per second it handles.

WHY. Before we can balance load we need a picture of load. The autoscaler cannot see "the model" — it only sees numbers coming off each box. So the whole derivation is built on: boxes (replicas) + a dial on each box (the metric).

PICTURE. Four identical boxes, each with a little pressure-gauge showing how busy it is.

Step 2 — The key assumption: load splits evenly

WHAT. We assume all incoming work is shared equally across the replicas. If total work is $L$ and there are $R$ boxes, each box carries $L / R$ .

WHY. A model server sits behind a Service that load-balances round-robin, so this "equal share" is almost true in practice. We need it because it is the one idea that lets us predict what happens when we add boxes: add a box, and each box's share shrinks.

PICTURE. A single pipe of total load $L$ splitting into $R$ equal streams.

Step 3 — Total load is a conserved quantity

WHAT. Rearrange Step 2. Multiply both sides by $R$ : $L = R \cdot M .$

WHY. This is the trick the whole formula rests on. The total work $L$ does not care how many boxes exist — the traffic arriving is set by your users, not by you. So $L$ is fixed at the instant we decide to scale. Whatever we do to $R$ , the product $R \cdot M$ must stay equal to that same $L$ . This is exactly like conservation of a quantity in a feedback system: something is preserved while its pieces trade off.

PICTURE. A see-saw: on one side "number of boxes $R$ ", on the other "busy-ness per box $M$ ". The see-saw's total weight $L = R \cdot M$ is a fixed block that never changes — only how it's distributed does.

Step 4 — Write the "now" and the "wish" states

WHAT. We name two moments:

Now: $R_{c u r}$ replicas, showing average metric $M_{c u r}$ .
Wish: $R_{d es}$ replicas, where we want each to show the target $M_{t a r}$ .

WHY. The autoscaler is a control loop: it compares observed against desired (the reconcile idea from the parent note). Here the "observed" is $M_{c u r}$ and the "desired per-box" is $M_{t a r}$ . We introduce $M_{t a r}$ because you pick it — e.g. "keep every box around 50% CPU" — as the comfortable operating point.

PICTURE. Two see-saws side by side: the left ("now") tilted heavy (boxes overloaded, $M_{c u r}$ high); the right ("wish") level at $M_{t a r}$ .

Step 5 — Set the two totals equal and solve

WHAT. Both states describe the same instant of traffic, so both have the same total load $L$ (Step 3). Write $L$ two ways and equate: $R_{c u r} \cdot M_{c u r} = L = R_{d es} \cdot M_{t a r} .$ Divide both sides by $M_{t a r}$ to isolate our unknown: $R_{d es} = R_{c u r} \cdot \frac{M _{c u r}}{M _{t a r}} .$

WHY. We want to know how many boxes make the gauges read $M_{t a r}$ instead of $M_{c u r}$ . Solving for $R_{d es}$ answers exactly that. The ratio $M_{c u r} / M_{t a r}$ is doing the real work: it says "how many times too busy are we right now?"

PICTURE. Same total block $L$ dropped onto a wider base — spread over more boxes so each sits lower, exactly at the $M_{t a r}$ line.

Step 6 — Why we must round UP (the ceiling)

WHAT. $R_{d es}$ from Step 5 can come out fractional (like $7.2$ ), but you cannot run $0.2$ of a box. The ceiling $⌈ x ⌉$ means "round up to the next whole number": $⌈ 7.2 ⌉ = 8$ , $⌈ 7.0 ⌉ = 7$ .

WHY up and not down? If we rounded $7.2$ down to $7$ , those 7 boxes would each carry more than the fair share for target — they'd end up above $M_{t a r}$ , i.e. still overloaded. Rounding up guarantees we never under-provision; the extra box just leaves everyone slightly below target, which is the safe side. This is the same "err toward safety" bias the parent note uses.

PICTURE. A number line at $7.2$ : an arrow pointing left to $7$ (labelled "boxes stay overloaded — bad") and right to $8$ ("everyone safely under target — chosen").

Step 7 — The dead-band: why K8s sometimes does nothing

WHAT. Real gauges jitter. If the computed ratio is very close to $1$ (default: within $10%$ , i.e. between $0.9$ and $1.1$ ), Kubernetes refuses to act and keeps $R_{c u r}$ .

WHY. Without this, a metric wobbling from $49%$ to $51%$ around a $50%$ target would make the autoscaler add and remove boxes endlessly — thrashing. A dead-band is hysteresis: exactly what a thermostat does so it doesn't click on/off every second (see Control Systems - Feedback Loops). It trades a little precision for a lot of stability, which also protects your bill (see Cost Optimization in Cloud ML — constant scaling churns machines).

PICTURE. The ratio number line with a shaded "no-action zone" from $0.9$ to $1.1$ around $1.0$ ; outside it, arrows for "scale up" (ratio $> 1.1$ ) and "scale down" (ratio $< 0.9$ ).

Step 8 — Edge & degenerate cases (never leave the reader stranded)

WHAT / WHY. Formulas must survive weird inputs. Walk each:

Definition Every corner of the formula

$M_{c u r} = 0$ (zero traffic). Ratio $= 0$ , raw $R_{d es} = 0$ . But $0 < 0.9$ so we'd scale down — Kubernetes clamps to a configured minReplicas (often 1), so a live service never drops to zero boxes.
$M_{t a r} = 0$ (target of zero). Division by zero — undefined, and meaningless ("keep every box at 0% busy" is impossible). K8s forbids a zero/empty target; you must set a positive one.
Ratio huge (traffic spike). $R_{d es}$ could jump 10×. A maxReplicas cap and a per-window scale-up rate limit stop a runaway that would blow your cloud budget.
Crash-looping replicas. Their metric may read low or garbage → the formula gives a wrong answer. HPA is for load, not health; readiness probes, not this formula, handle broken Pods (parent-note mistake #4).
Non-linear load (batched GPU inference). Our whole derivation assumed load splits linearly (Step 2). With request batching, doubling replicas may not halve latency — the linear see-saw bends, and $R_{d es}$ is only an estimate. Good targets are chosen with headroom for this.

PICTURE. A small map of the ratio axis marking the clamps: minReplicas floor on the left, dead-band in the middle, maxReplicas ceiling on the right — the "safe operating corridor".

The one-picture summary

Everything at once: measured gauges → average $M_{c u r}$ → compare to $M_{t a r}$ → ratio → ceiling → clamp → new replica count, looped forever.

Recall Feynman retelling — explain the whole walkthrough to a 12-year-old

Imagine 4 identical chefs each with a "tiredness dial". You want every chef about half-tired (that's your target). Right now they read 90% — way too tired, because the total pile of orders is fixed but there aren't enough chefs to share it. Total work stays the same whether you have 4 chefs or 8; only how tired each one gets changes. So you ask: "how many times over comfort are we?" — 90 ÷ 50 = 1.8 times. Multiply your chefs by 1.8 → 7.2 chefs. You can't have 0.2 of a chef, and rounding down leaves everyone still overworked, so you round up to 8. Now the same pile spread over 8 chefs leaves each near half-tired — perfect. And if the dials only wobble a tiny bit (say they read 48 when you wanted 50), you do nothing at all, because hiring and firing chefs over a 2% jiggle would be silly. That "do nothing near the target" rule is the same reason a heater doesn't click on and off every second. That's the whole autoscaler.

Recall Rebuild the formula from memory

Start from load splits evenly ::: $M = L / R$ , so $L = R \cdot M$ (total load conserved). Set now-total = wish-total ::: $R_{c u r} M_{c u r} = R_{d es} M_{t a r}$ . Solve for the unknown ::: $R_{d es} = R_{c u r} M_{c u r} / M_{t a r}$ . Fix the "no half a box" problem ::: round up with the ceiling $⌈ \cdot ⌉$ . Fix the thrashing problem ::: ignore changes inside the ±10% dead-band. Fix the extremes ::: clamp between minReplicas and maxReplicas.

Connections

Kubernetes for ML workloads — the parent; this page derives its HPA result.
Control Systems - Feedback Loops — the dead-band = hysteresis; reconcile = feedback.
Horizontal vs Vertical Scaling — HPA is horizontal (more boxes), not bigger boxes.
Model Serving (TF-Serving, TorchServe, Triton) — the servers being replicated.
Cost Optimization in Cloud ML — why min/max caps and dead-bands protect the bill.