Kubernetes for ML workloads: Exercises — AI-ML deep dive

Level 1 — Recognition

Exercise 1.1. For each requirement, name the single Kubernetes object that fits best. (a) Run a nightly retraining job at 02:00. (b) Serve a stateless inference model behind one stable address. (c) Give each distributed-training worker a fixed name like worker-0 and its own disk. (d) Keep a dataset alive even after the Pod using it is deleted.

Recall Solution 1.1

(a) CronJob — a Job on a schedule. (b) A Deployment (runs the replicas) fronted by a Service (the stable address). The Service is what gives the one stable address. (c) StatefulSet — the only object that guarantees stable network identity plus a persistent volume per replica. (d) PersistentVolumeClaim (PVC) — storage whose lifetime is decoupled from any one Pod. Why: each object exists to solve exactly one class of problem — match the lifecycle (run-once? scheduled? stateful? stateless?) to the object.

Exercise 1.2. Which field does the scheduler read to decide where a Pod goes: request or limit? What does the other field control?

Recall Solution 1.2

The scheduler reads request — that is the guaranteed minimum it reserves on a node, so it is the placement number. The limit is the runtime ceiling: exceed CPU → the container is throttled; exceed memory → the Pod is OOM-killed (Out-Of-Memory killed by the Linux kernel).

Level 2 — Application

Exercise 2.1. A node has $C = 32$ GB allocatable RAM. It already runs three Pods requesting 6 GB, 6 GB, and 10 GB. You submit a new Pod requesting $r = 8$ GB. Does it fit? If yes, how much RAM is left afterward?

Recall Solution 2.1

Sum of existing requests: $6 + 6 + 10 = 22$ GB. Free capacity $= C - \sum_{i} r_{i} = 32 - 22 = 10$ GB. Fit test: $r \leq 10$ ? We need $8 \leq 10$ → true, it fits. Remaining after placement: $10 - 8 = 2$ GB. Why the fit test and not "does total ≤ C"? They are the same inequality rearranged: $r \leq C - \sum r_{i}$ is just $\sum r_{i} + r \leq C$ . The subtraction form makes the scheduler's question — "is there room right now?" — visible directly.

Exercise 2.2. A cluster has $N = 5$ nodes, each with $C = 40$ GB. Each inference replica requests $r = 12$ GB. What is the maximum number of replicas the cluster can hold? How much RAM is stranded per node?

Recall Solution 2.2

Per node: $⌊ 40/12 ⌋ = ⌊ 3.33 \dots ⌋ = 3$ replicas, using $3 \times 12 = 36$ GB. Across the cluster: $N ⌊ C / r ⌋ = 5 \times 3 = 15$ replicas. Stranded per node: $40 - 36 = 4$ GB (less than one 12 GB request, so unusable). Total stranded $= 5 \times 4 = 20$ GB. Why floor per node, not globally? A Pod cannot be split across two machines. If you (wrongly) computed $⌊ 5 \cdot 40/12 ⌋ = ⌊ 16.6 ⌋ = 16$ , you'd be assuming the leftover 4 GB scraps from different nodes could be glued together — they can't. See the figure below.

Exercise 2.3. You currently run $R_{c u r} = 3$ replicas. Average CPU is $M_{c u r} = 80%$ ; your target is $M_{t a r} = 40%$ . What replica count does the HPA request?

Recall Solution 2.3

$R_{d es} = ⌈ 3 \cdot 80/40 ⌉ = ⌈ 3 \cdot 2 ⌉ = ⌈ 6 ⌉ = 6$ . Why it doubles: the ratio $M_{c u r} / M_{t a r} = 2$ says each replica is running at twice the target load. Under the linear-sharing assumption, doubling the replica count halves per-replica load, dragging 80% down to 40%.

Level 3 — Analysis

Exercise 3.1. A node has 8 GB. You must choose a per-replica request. Compare $r = 3$ GB versus $r = 2$ GB: how many replicas fit each, how much is stranded, and which is the better packing choice?

Recall Solution 3.1

$r = 3$ : $⌊ 8/3 ⌋ = 2$ replicas, using 6 GB, 2 GB stranded. $r = 2$ : $⌊ 8/2 ⌋ = 4$ replicas, using 8 GB, 0 GB stranded (2 divides 8 exactly). $r = 2$ is the better packer: choosing a request that divides node capacity evenly eliminates stranded waste. This is why teams often size requests as clean fractions of the node (e.g. 2, 4, 8 GB on a 16 GB node). Why analyse the divisor and not just "smaller is better"? Smaller requests aren't automatically better — a 2.5 GB request also gives $⌊ 8/2.5 ⌋ = 3$ Pods with 0.5 GB stranded. It is divisibility, not smallness, that kills waste.

Exercise 3.2. The HPA sees $R_{c u r} = 8$ , $M_{c u r} = 52%$ , $M_{t a r} = 50%$ . The default HPA tolerance is 10% (it does nothing while the ratio $M_{c u r} / M_{t a r}$ is within $[0.9, 1.1]$ ). Does it scale? Now the metric climbs to $M_{c u r} = 60%$ — does it scale, and to what?

Recall Solution 3.2

First case: ratio $= 52/50 = 1.04$ . Since $0.9 \leq 1.04 \leq 1.1$ , it is inside the dead-band → no action. Second case: ratio $= 60/50 = 1.2 > 1.1$ → outside the band, so HPA acts. $R_{d es} = ⌈ 8 \cdot 60/50 ⌉ = ⌈ 9.6 ⌉ = 10$ . Why a dead-band exists: without it, a ratio of 1.04 would demand $⌈ 8 \cdot 1.04 ⌉ = ⌈ 8.32 ⌉ = 9$ , then a dip to 0.96 would drop back to 8, then up again — endless thrash. The 10% tolerance is hysteresis, exactly like a thermostat that waits for a 1° gap before firing.

Level 4 — Synthesis

Exercise 4.1. You must serve a model under these constraints: nodes have 4 GPUs and 64 GB RAM each; each replica requests nvidia.com/gpu: 1 and 20 GB RAM. What is the true binding constraint per node — GPU or RAM — and how many replicas fit one node? With 3 such nodes, how many replicas total?

Recall Solution 4.1

Compute the two independent limits, then take the minimum:

GPU-bound: $⌊ 4/1 ⌋ = 4$ replicas.
RAM-bound: $⌊ 64/20 ⌋ = ⌊ 3.2 ⌋ = 3$ replicas. The node can only honour both requests simultaneously, so per-node capacity $= min (4, 3) = 3$ . RAM is the binding constraint. The 4th GPU sits idle. Cluster total: $N \times 3 = 3 \times 3 = 9$ replicas — leaving 3 GPUs stranded (one per node). Why min of the two? A Pod needs a whole bundle (1 GPU and 20 GB) — the scheduler can only place as many bundles as the scarcest resource allows. To unstick the wasted GPUs you'd cut RAM per replica to $\leq 16$ GB so $⌊ 64/16 ⌋ = 4 =$ GPU count.

Exercise 4.2. Design the QoS for a GPU inference Deployment. For memory and GPU, should request equal limit? For CPU, should they? Justify each using the OOM / throttle / over-commit rules.

Recall Solution 4.2

GPU: request == limit == 1. GPUs cannot be over-committed in stock Kubernetes — you get whole, exclusive GPUs, so the two numbers are forced equal.
Memory: request == limit. Exceeding a memory limit means OOM-kill; matching them removes the risky over-commit gap and gives predictable behaviour under load (the "Guaranteed" QoS class).
CPU: request < limit is fine. Exceeding a CPU limit only throttles (slows) the container — it does not kill it. So requesting the typical CPU and allowing a higher limit lets a Pod burst during traffic spikes while packing more Pods per node when idle. Why treat CPU differently? The failure modes differ: memory over-run = death (irreversible), CPU over-run = slowdown (reversible). You only tolerate over-commit where the penalty is survivable.

Level 5 — Mastery

Exercise 5.1. Full stack design. A latency-critical inference service must:

hold at least 6 replicas at all times (min replicas),
scale up on CPU with target $M_{t a r} = 60%$ ,
run only on GPU nodes (4 GPUs, 48 GB RAM each), each replica requesting 1 GPU and 10 GB RAM.

You have a cluster of 4 GPU nodes. (a) What is the cluster's hard replica ceiling? (b) During a spike, 6 replicas report average CPU $M_{c u r} = 95%$ . What does the HPA request, and can the cluster satisfy it? (c) If not fully satisfiable, what two levers fix it, and what does each cost?

Recall Solution 5.1

(a) Ceiling. Per node take the min over resources: GPU: $⌊ 4/1 ⌋ = 4$ ; RAM: $⌊ 48/10 ⌋ = ⌊ 4.8 ⌋ = 4$ . Per-node capacity $= min (4, 4) = 4$ . Cluster ceiling $= 4 nodes \times 4 = 16$ replicas. (b) HPA demand. $R_{d es} = ⌈ 6 \cdot 95/60 ⌉ = ⌈ 9.5 ⌉ = 10$ . $10 \leq 16$ (the ceiling), and $10 \geq 6$ (the min), so yes — the cluster schedules all 10. Every replica gets 1 GPU (10 of the 16 available) and 10 GB (100 of 192 GB). It fits. (c) If the ceiling were exceeded (say demand were 20): two levers —

Horizontal scaling of nodes (add more GPU nodes via the Cluster Autoscaler): each new node adds 4 replica slots. Cost: more GPU node-hours = higher cloud bill, and cold-start latency while nodes boot.
Shrink per-replica GPU footprint (MIG / time-slicing): pack more replicas per GPU. Cost: contention — replicas share a GPU, so per-request latency can rise, hurting a latency-critical service. Why both levers and not one? Adding nodes buys guaranteed isolation but costs money; sharing GPUs buys density but costs latency. The latency-critical requirement biases you toward adding nodes here. Why check both ceiling and min? The scheduled count must satisfy $min \leq R_{d es} \leq ceiling$ . HPA clamps its request into that window — it will never drop below 6 nor exceed 16.

Exercise 5.2. Reason about the control loop. Your HPA holds a service near 60% CPU. A bad image: model:v3 is deployed and every new Pod crash-loops (starts, fails, restarts). CPU on the healthy remaining Pods rises to 95% because they absorb all traffic. What does the HPA do, does it help, and what is the correct fix?

Recall Solution 5.2

HPA sees $M_{c u r} = 95%$ vs target 60% and requests more replicas: $⌈ R_{c u r} \cdot 95/60 ⌉$ . But the new replicas use the broken image, so they crash-loop too — adding count does nothing for a health problem. Worse, a readinessProbe should keep crash-looping Pods out of the Service's load-balancing pool, so traffic keeps piling onto the few healthy Pods. Correct fix: roll back the Deployment to model:v2 (health/rollout problem), and rely on livenessProbe/readinessProbe — not HPA — for health. HPA scales for load; probes manage health. Conflating them is the classic error. Why the reconciliation loop can't save you here: the loop faithfully drives actual toward desired ("run N copies of model:v3"). If the desired state itself is broken, reconciliation obediently reproduces the breakage N times. The loop is only as good as the desired state you declare.

Active recall

Recall Rapid-fire (hide answers)

Fit test inequality? ::: $r \leq C - \sum_{i} r_{i}$ Max replicas across N nodes? ::: $N ⌊ C / r ⌋$ HPA desired-replica formula? ::: $⌈ R_{c u r} M_{c u r} / M_{t a r} ⌉$ Why floor per node, not globally? ::: A Pod can't split across machines; per-node scraps can't be glued. Binding constraint when several resources are requested? ::: The minimum of the per-resource floors. Should GPU request equal its limit? ::: Yes — GPUs can't be over-committed. Should CPU request equal its limit? ::: No — CPU over-run only throttles, so bursting is safe. Does HPA fix a crash-looping deploy? ::: No — that's a probe/rollback job, not a scaling job.

Connections

Kubernetes for ML workloads — parent topic these exercises drill.
Horizontal vs Vertical Scaling — adding nodes vs bigger nodes (Ex 5.1c).
Cost Optimization in Cloud ML — the price of over-provisioning (Ex 5.1, L5 trap).
Control Systems - Feedback Loops — the HPA dead-band as hysteresis (Ex 3.2).
Model Serving (TF-Serving, TorchServe, Triton) — what runs inside these replicas.
Distributed Training — StatefulSet ranks (Ex 1.1c).
CI-CD for ML — the rollout/rollback flow behind Ex 5.2.