Kubernetes for ML workloads: Worked examples — AI-ML deep dive

Before any formula: read every symbol out loud in plain words so nothing enters undefined.

The scenario matrix

Every cell below is a distinct shape of problem. The worked examples are tagged with the cell(s) they cover.

#	Cell (case class)	What makes it tricky	Example
A	Exact fit — request divides capacity	no waste; floor is exact	Ex 1
B	Leftover / stranded — request does not divide capacity	floor drops a fraction → wasted resource	Ex 2
C	Two constraints at once — RAM and GPU both limit	the tighter one wins	Ex 3
D	HPA scale-up — metric above target	ratio $> 1$ , ceil rounds up	Ex 4
E	HPA scale-down — metric below target	ratio $< 1$ , replicas shrink	Ex 5
F	Degenerate: zero load — $M_{c u r} = 0$	formula wants 0 replicas → floor/min-replica clamp	Ex 6
G	Dead-band / no-op — ratio inside tolerance	correct action is do nothing	Ex 7
H	Limiting value: hits cluster ceiling — desired > max that fits	HPA blocked by capacity, not maths	Ex 8
I	Word problem — real cost/traffic story end-to-end	translate English → symbols	Ex 9
J	Exam twist — the "obvious" answer is the trap	crash-loop / QoS misconception	Ex 10

Group 1 — Packing Pods onto nodes (cells A, B, C)

Worked example Ex 1 — Exact fit (cell A)

A node has $C = 16$ GB RAM. Each inference Pod requests $r = 4$ GB. How many Pods fit on one node, and how much RAM is left over?

Forecast: guess the Pod count and the leftover before reading on.

Compute the raw ratio $C / r$ . $16/4 = 4$ . Why this step? This is "how many request-sized slots fit end-to-end". It's a division because we're repeatedly subtracting one request until the node is full.
Apply the floor. $⌊ 4 ⌋ = 4$ . Look at the left panel of the figure: four equal mint blocks tile the bar with no gap. Why this step? Floor drops any fractional Pod. Here the ratio is already a whole number, so floor changes nothing — that is the definition of "exact fit".
Leftover $= C - (Pods \times r) = 16 - 4 \times 4 = 0$ GB. Why this step? Leftover is the RAM no Pod claimed. Zero here means perfect packing.

Verify: $4 \times 4 = 16 = C$ . Every GB is used, no Pod exceeds the wall. ✓

Worked example Ex 2 — Leftover / stranded resource (cell B)

Same node, $C = 16$ GB, but now each Pod requests $r = 5$ GB. Pods and stranding?

Forecast: does a 4th Pod fit? How much is wasted?

Raw ratio. $16/5 = 3.2$ . Why this step? 3.2 "slots" means three whole Pods plus a fractional remainder — the remainder is the warning sign of waste.
Floor. $⌊ 3.2 ⌋ = 3$ . Right panel of the figure: three coral blocks, then a butter-yellow stub that is too short to hold a 4th Pod. Why this step? You physically cannot schedule 0.2 of a Pod. Floor is what turns "3.2 slots" into "3 Pods that actually run".
Stranded RAM $= 16 - 3 \times 5 = 16 - 15 = 1$ GB. Why this step? This 1 GB is smaller than one request (5 GB), so no Pod can ever use it → it is stranded. This is exactly why the parent note says: choose request sizes that divide node capacity.

Verify: $3 \times 5 = 15 \leq 16$ (fits) and $4 \times 5 = 20 > 16$ (a 4th does not). Stranded $1 < 5$ , so it is truly unusable. ✓

Worked example Ex 3 — Two constraints, tighter one wins (cell C)

A GPU node has $C_{R A M} = 64$ GB and $C_{GP U} = 4$ whole GPUs. Each training Pod requests $r_{R A M} = 10$ GB and $r_{GP U} = 1$ . How many Pods fit?

Forecast: which resource runs out first — RAM or GPU?

Pods allowed by RAM alone. $⌊ 64/10 ⌋ = ⌊ 6.4 ⌋ = 6$ . Why this step? Pretend GPUs are infinite; RAM alone would allow 6 Pods.
Pods allowed by GPU alone. $⌊ 4/1 ⌋ = 4$ . Why this step? Now pretend RAM is infinite; GPU alone allows 4 Pods. GPUs are indivisible (parent note), so the floor is exact.
Take the minimum. $min (6, 4) = 4$ Pods. Why this step? A Pod needs both resources simultaneously. The scheduler can only place as many as the scarcest resource permits — here GPUs. This "min over all resources" is the real fit rule; single-resource formulas are just the easy case.

Verify: With 4 Pods: RAM used $= 40 \leq 64$ ✓, GPU used $= 4 \leq 4$ ✓. A 5th Pod needs a 5th GPU that does not exist. ✓ Note 24 GB of RAM is stranded — a classic sign your node type is GPU-bound, not RAM-bound.

Group 2 — Autoscaling with the HPA formula (cells D, E, F, G, H)

Recall the engine, and re-earn every symbol:

Worked example Ex 6 — Degenerate: zero load (cell F)

$R_{c u r} = 5$ , $M_{c u r} = 0%$ (no traffic at 3 a.m.), $M_{t a r} = 50%$ . What does HPA do?

Forecast: does it scale to 0?

Ratio. $0/50 = 0$ . Why this step? Zero measured load → the formula literally wants zero replicas.
Scale + ceil. $⌈ 5 \times 0 ⌉ = ⌈ 0 ⌉ = 0$ . Why this step? Mathematically clean, but a Deployment with 0 replicas serves nothing and can't measure future traffic to wake up.
Clamp to minReplicas. HPA never goes below its configured floor (default $\geq 1$ ; you set e.g. minReplicas: 2). So the real answer is 2, not 0. Why this step? The formula is only half the rule — minReplicas/maxReplicas bound it. Zero-scaling requires special tooling (KEDA), not stock HPA. See Cost Optimization in Cloud ML for scale-to-zero patterns.

Verify: $max (minReplicas, R_{d es}) = max (2, 0) = 2$ ✓. Formula alone lies; the clamp is the truth.

Worked example Ex 8 — Limiting value: desired exceeds what fits (cell H)

$R_{c u r} = 6$ , $M_{c u r} = 95%$ , $M_{t a r} = 40%$ , each replica requests 1 GPU. HPA wants how many, and can the cluster deliver if it has 2 nodes × 4 GPUs?

Forecast: will every requested replica actually run?

HPA target. $⌈ 6 \times 95/40 ⌉ = ⌈ 14.25 ⌉ = 15$ . Why this step? Pure demand-side maths: to drag 95% down to 40% needs 15 replicas.
Cluster GPU ceiling. Max GPU-Pods $= N ⌊ C / r ⌋ = 2 \times ⌊ 4/1 ⌋ = 8$ . Why this step? This is the supply side — the packing formula from Group 1. It ignores traffic entirely.
Reconcile the two. HPA wants 15 but only $min (15, maxReplicas, 8) = 8$ can be Scheduled. The other 7 sit Pending with Insufficient nvidia.com/gpu. Why this step? HPA computes a desire; the scheduler enforces reality. When desire > capacity, the extra Pods don't crash — they wait. The fix is more nodes (Cluster Autoscaler) — see Horizontal vs Vertical Scaling.

Verify: $8 \leq 15$ so demand isn't met by scheduling alone. Running 8 replicas gives per-replica load $= 6 \times 95/8 = 71.25% > 40%$ — still hot, confirming you are capacity-bound, not autoscaler-bound. ✓

Group 3 — Word problem & exam twist (cells I, J)

Worked example Ex 9 — Full word problem (cell I)

Your Triton server handles 200 requests/sec (RPS) per replica comfortably (that's your $M_{t a r}$ ). At noon, 6 replicas together are seeing 330 RPS each on average ( $M_{c u r}$ ). Each replica requests 1 GPU. Your cluster is 3 nodes × 4 GPUs. (a) How many replicas does HPA request? (b) Do they all fit? (c) If a GPU-node costs $3/hr and idle GPUs still bill, how many GPUs are wasted at the new count?

Forecast: guess the replica count and whether the cluster copes.

(a) Translate to symbols. $R_{c u r} = 6$ , $M_{c u r} = 330$ , $M_{t a r} = 200$ . $R_{d es} = ⌈ 6 \times 330/200 ⌉ = ⌈ 9.9 ⌉ = 10$ . Why this step? RPS is a per-replica rate that scales linearly with replica count — exactly the HPA assumption, so the formula applies unchanged.
(b) Capacity check. Max GPU-Pods $= 3 \times ⌊ 4/1 ⌋ = 12 \geq 10$ . ✓ All 10 fit. Why this step? Compare desire (10) to supply (12) using the packing formula. Since $10 \leq 12$ , no Pending Pods.
(c) Waste accounting. 10 GPUs used of 12 available → 2 GPUs idle → $2\times\$ 3 = $6/\text{hr}$ burnt. Why this step? Idle-but-reserved GPUs still bill. This is the Cost Optimization in Cloud ML tension: keep headroom for bursts vs. pay for idleness.

Verify: load per replica after scaling $= 6 \times 330/10 = 198 RPS \leq 200$ ✓ (under target, good). $10 \leq 12$ nodes-capacity ✓. Idle $= 12 - 10 = 2$ ✓.

Worked example Ex 10 — Exam twist: the trap answer (cell J)

A GPU inference Pod is set with requests.memory = limits.memory = 8Gi, requests.cpu = 2, limits.cpu = 4. Under a traffic spike the Pod's readiness probe starts failing and it crash-loops. An engineer says: "Bump maxReplicas on the HPA — autoscaling will absorb the load." Is that the fix? What actually happens, and what is the Pod's QoS class?

Forecast: will more replicas help a crash-looping Pod?

Diagnose HPA's job. HPA multiplies replica count by a load ratio. It reads a metric; it does not read health. Why this step? Separating concerns: HPA = how many, probes = is each one alive. Raising maxReplicas just lets a broken image crash in more copies (parent note, mistake #4).
Right lever. A failing readiness probe means the container isn't serving; fix the image/OOM/startup, or tune readinessProbe/livenessProbe. Scaling count is orthogonal. Why this step? Because requests.memory == limits.memory == 8Gi, a spike that needs >8 GB triggers an OOM-kill, not a bump — the ceiling is hard.
QoS class. With every resource having request == limit for memory (and requests set for CPU), and no unset resource, the Pod is Guaranteed QoS. Why this step? Kubernetes assigns Guaranteed only when for every container each resource's request equals its limit. (CPU request 2 ≠ limit 4 means CPU can burst, but Guaranteed classification hinges on memory matching here — the memory 8Gi==8Gi is what buys predictability; the true Guaranteed rule needs CPU equal too, so strictly this Pod is Burstable because CPU differs.) Correction earned: since $cpu request = 2 \neq = cpu limit = 4$ , not all resources match ⇒ QoS class is Burstable, not Guaranteed. This is the twist: matching memory alone is not enough.

Verify: HPA fix rejected (health ≠ load) ✓. QoS: not every resource has request==limit (CPU 2≠4) ⇒ Burstable ✓. See CI-CD for ML — a crash-looping image should have been caught before deploy.

Active recall

Recall Test the matrix (hide answers)

Node 20 GB, Pod 6 GB — Pods and stranded? ::: $⌊ 20/6 ⌋ = 3$ , stranded $20 - 18 = 2$ GB.
Two limits RAM→7, GPU→4, which wins? ::: $min (7, 4) = 4$ ; the scarcer resource.
$R_{c u r} = 3$ , $M_{c u r} = 80$ , $M_{t a r} = 40$ ? ::: $⌈ 3 \cdot 2 ⌉ = 6$ .
$M_{c u r} = 0$ — real answer? ::: minReplicas, not 0 (stock HPA can't scale to zero).
Ratio 0.95 with 10% tolerance? ::: inside dead-band → do nothing.
HPA wants 15 but only 8 GPUs fit? ::: 8 run, 7 Pending; add nodes.
Fix for a crash-looping Pod? ::: probes / image, NOT more replicas.

Connections

Kubernetes for ML workloads — parent: the three engines these examples exercise.
Horizontal vs Vertical Scaling — Ex 8's "add nodes" is horizontal cluster scaling.
Cost Optimization in Cloud ML — stranded RAM (Ex 2) and idle GPUs (Ex 9) are pure waste.
Model Serving (TF-Serving, TorchServe, Triton) — the RPS servers in Ex 9.
Control Systems - Feedback Loops — the HPA dead-band (Ex 7) is hysteresis.
Distributed Training — GPU packing (Ex 3) governs how many worker ranks fit.
CI-CD for ML — catch the broken image of Ex 10 before it ships.