Noesis
5.3.9 · D5MLOps & Deployment

Question bank — Kubernetes for ML workloads

1,679 words8 min read↑ Back to topic

Before the traps, one shared vocabulary reminder so every item below is readable from line one:

True or false — justify

Every item is a claim. Decide true/false, then give the because.

A Deployment with replicas: 3 guarantees exactly 3 Pods are serving traffic at all times.
False — it targets 3 running Pods, but during a rolling update or a crash there can be temporarily 2 or 4; and a Pod can be running yet failing its readinessProbe, so it is alive but not receiving traffic.
Deleting a Pod that a Deployment owns permanently reduces your capacity by one.
False — the ReplicaSet notices actual < desired and creates a replacement almost immediately; that is the reconciliation loop doing its job.
A Service load-balances to the specific Pod IPs you listed when you created it.
False — a Service selects Pods by label selector, not by IP; Pods die and get new IPs constantly, and the Service tracks whatever currently matches the labels.
If you set no memory limit, a Pod can safely grow because it "borrows" spare RAM.
False — with no limit a leaking Pod can consume the whole Node's RAM, and the kernel then OOM-kills neighbouring Pods too, widening the blast radius beyond the offender.
Setting request = limit = peak usage is the safest configuration.
False for cost — the scheduler reserves the request even while the Pod idles, so you pay for peak 24/7 and pack far fewer Pods per node; it is only correct for resources that cannot be over-committed (GPU, and memory if you want Guaranteed QoS).
The HPA can rescue a crash-looping Pod by adding replicas.
False — HPA scales count from a load metric; a broken image just crash-loops in more copies. Health is the job of livenessProbe/readinessProbe, not the autoscaler.
A CPU limit being exceeded kills the Pod, just like memory.
False — CPU is compressible: exceeding the CPU limit only throttles (slows) the Pod. Memory is incompressible, so exceeding the memory limit triggers an OOM-kill.
A Job and a Deployment are interchangeable for training.
False — a Deployment tries to keep Pods running forever (restarting a finished trainer endlessly), while a Job runs to completion and stops once it succeeds once, which is exactly what batch training wants.
nvidia.com/gpu: 0.5 lets two Pods share one GPU in stock Kubernetes.
False — stock K8s treats GPUs as indivisible whole units; fractional requests are invalid unless you enable MIG or time-slicing extensions.
Two replicas of the same Deployment can be scheduled onto the same Node.
True — nothing forbids co-location; the scheduler only checks that free capacity ≥ request. To force spread you need anti-affinity rules or topology spread constraints.

Spot the error

Each line contains a subtly wrong statement. Say what's wrong.

"I gave my Pod requests: {memory: 8Gi} and limits: {memory: 4Gi} for safety."
The limit is below the request, which is invalid — a limit is a ceiling and must be ≥ the request, otherwise the reserved minimum already violates the maximum.
"My cluster has 3 nodes × 12 GB = 36 GB free, so a 10 GB Pod and a 10 GB Pod and a 10 GB Pod all fit."
Wrong reasoning — capacity is packed per node, not pooled. Each 10 GB Pod needs a whole node (12 GB each holds one), so three fit here, but a fourth would fail even though 6 GB total remains — it's stranded across nodes.
"I named my Pods worker-0, worker-1 in a Deployment so distributed training has stable ranks."
A Deployment gives Pods random suffixes and unstable identity; stable ordinal names (worker-0) that survive restarts require a StatefulSet.
"HPA read 48% CPU against a 50% target and scaled down to save money."
No action occurs — a ratio of 0.96 sits inside the default ~10% tolerance dead-band, so HPA deliberately does nothing to avoid thrashing.
"Requests are the ceiling; limits are what the scheduler reserves."
Reversed — the request is reserved for placement; the limit is the ceiling for runtime enforcement. Mnemonic: "Request to Reserve, Limit to Lockdown."
"My GPU Pods keep landing on CPU-only nodes; I'll add more replicas."
More replicas won't change where they land. You need nodeSelector (or taints/tolerations) to force GPU Pods onto GPU nodes.
"I set replicas: 3 and an HPA min=2 max=10 — so I always get at least 3."
Once an HPA attaches, it owns the replica count and can drop you to its min (2); the static replicas field is no longer authoritative.

Why questions

Answer the because, not just the what.

Why does the scheduler use request and not limit for placement?
Because placement must reserve a guaranteed floor of resources; if it packed by the (larger, optimistic) limit it might either waste capacity or, packing by nothing, over-subscribe a node and cause OOM-kills.
Why does the HPA formula round up (ceil) instead of to nearest?
Rounding up guarantees you never under-provision — it is safer to run one extra replica than to leave load per replica above target and drop requests.
Why is the max-replicas formula and not ?
Because a Pod cannot be split across two machines; resources pack node-by-node, so leftover slivers on each node () are stranded and cannot be summed to host another Pod.
Why does matching request = limit give the strongest ("Guaranteed") QoS class?
When they are equal the Pod's usage envelope is fully reserved and fully capped, so Kubernetes never needs to evict it under pressure — it is the last to be killed.
Why is the reconciliation loop analogous to a thermostat?
Both continuously compare a target (desired temperature / desired replicas) with a measurement (actual temperature / actual Pods) and act to shrink the gap — a classic feedback control loop, see Control Systems - Feedback Loops.
Why does the HPA use a tolerance dead-band at all?
Without hysteresis, tiny metric wiggles around the target would trigger endless scale-up/scale-down oscillation; the dead-band absorbs noise so the system settles.
Why prefer a Service DNS name over a Pod IP in your client code?
Pod IPs are ephemeral — Pods die and respawn with new addresses — while the Service name and virtual IP are stable and continuously re-point to whatever Pods currently match the labels.

Edge cases

The scenarios the happy path never shows you.

What does the HPA formula give when (zero traffic)?
in raw math, but HPA clamps to its configured minReplicas, so you keep a warm floor instead of scaling to zero (unless a scale-to-zero autoscaler like KEDA is used).
A Pod requests 5 GB on a 16 GB node already holding 12 GB of requests — does it fit?
No: fit requires GB, and , so it stays Pending until space frees or another node has room.
What happens to a Job's Pod if the process exits with a non-zero code?
The Job restarts it (up to backoffLimit) because a Job's success condition is completing successfully once; repeated failures eventually mark the Job as Failed.
A single Pod requests more memory than any node can offer — what state is it in?
It stays permanently Pending (Unschedulable); no bin-packing or autoscaling of Pods helps, because the unit itself exceeds a whole node — you need a bigger node type (vertical, see Horizontal vs Vertical Scaling).
Node = 4 GPUs, three training Pods request 1 GPU each — how many GPUs are usable by a fourth Pod needing 2?
One GPU remains free, but the fourth Pod needs 2, so it cannot be placed — indivisible whole-GPU packing strands the single leftover.
What happens to a StatefulSet Pod's data when the Pod is deleted and recreated?
The PersistentVolumeClaim (and its data) survives and is re-attached to the same ordinal Pod (worker-0 reclaims worker-0's volume) — that persistence is the whole point of a StatefulSet over a Deployment.
If you scale a Deployment to replicas: 0, is it deleted?
No — the Deployment object and its config persist; it simply runs zero Pods, so you can scale back up instantly without re-applying YAML.

Connections