Kubernetes for ML workloads

WHY does ML even need Kubernetes?

WHAT is the core idea? You describe the end state in YAML. K8s runs a reconciliation loop: continuously compare desired state vs actual state, and take actions to close the gap.

$\text{action} = \text{reconcile}(\text{desired}, \text{observed})$

This is exactly a control system — like a thermostat. That analogy is the whole design.

The object hierarchy (build it from scratch)

HOW they nest: Deployment → ReplicaSet → Pods → Containers, all scheduled onto Nodes, exposed by a Service.

Deriving resource requests: the scheduler's arithmetic

Derivation — will a Pod fit? Let a Node have allocatable resource $C$ (e.g. 16 GB RAM). Existing Pods request $\sum_i r_i$ . A new Pod with request $r$ fits iff:

$r \le C - \sum_i r_i$

Derivation — how many replicas fit a cluster? With $N$ identical nodes each of capacity $C$ , and each replica requesting $r$ :

$\text{max replicas} = N \left\lfloor \frac{C}{r} \right\rfloor$

We use $\lfloor \cdot \rfloor$ per node, not globally, because a single Pod can't be split across two machines — resources are packed node-by-node.

Autoscaling: deriving the HPA formula

Derivation from a conservation idea. Suppose current replicas $R_{cur}$ collectively show metric value $M_{cur}$ (e.g. average CPU% or requests/sec), and you want each replica to sit at target $M_{tar}$ . Assume load is shared evenly and scales linearly with replicas. Total load $L = R_{cur}\cdot M_{cur}$ . To make per-replica load equal $M_{tar}$ :

$R_{cur}\cdot M_{cur} = R_{des}\cdot M_{tar} \;\Rightarrow\; R_{des} = R_{cur}\cdot \frac{M_{cur}}{M_{tar}}$

Kubernetes rounds up (ceil) so you never under-provision:

ML-specific patterns

Worked example Minimal GPU inference Deployment (YAML skeleton)

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: model-server
        image: myrepo/model:v2
        resources:
          requests: { cpu: "2", memory: "8Gi", nvidia.com/gpu: 1 }
          limits:   { cpu: "4", memory: "8Gi", nvidia.com/gpu: 1 }

Why request == limit for GPU/memory? GPUs can't be over-committed, and memory over-commit risks OOM-kills — matching them gives predictable, "Guaranteed" QoS.

Common mistakes (Steel-manned)

Active recall

Recall Test yourself (hide answers)

What are the two states K8s reconciles? → desired vs actual/observed.
Which field drives scheduling, request or limit? → request.
Formula for desired replicas? → $\lceil R_{cur} M_{cur}/M_{tar}\rceil$ .
What happens on exceeding a memory limit? → Pod is OOM-killed.
Which object runs batch training to completion? → Job.
Why does a Service use labels not IPs? → Pod IPs change constantly.

Recall Feynman: explain to a 12-year-old

Imagine you run a pizza shop with many identical chefs (Pods). You tell the manager (Kubernetes): "always keep 3 chefs working, each needs one oven." If a chef quits, the manager hires a new one instantly. If a huge rush comes, the manager adds chefs (autoscaling). Customers phone one number (Service) and the manager routes them to whichever chef is free — they never need to know which chef. You never manage chefs by hand; you just tell the manager the goal, and it keeps things true.

Connections

Docker & Containerization — Pods run containers; K8s orchestrates them.
Model Serving (TF-Serving, TorchServe, Triton) — what lives inside the inference Pod.
Horizontal vs Vertical Scaling — HPA is horizontal.
CI-CD for ML — rolling updates deploy new model versions.
Distributed Training — StatefulSets + PVCs for multi-worker ranks.
Cost Optimization in Cloud ML — bin-packing & request-sizing.
Control Systems - Feedback Loops — the reconcile loop is a controller.

What mental model best describes Kubernetes?

A cluster OS running a reconciliation loop that keeps actual state equal to a declared desired state.

Which resource field does the scheduler use to place a Pod?

The request (guaranteed reserved minimum), not the limit.

What happens when a Pod exceeds its memory limit vs CPU limit?

Memory over-limit → Pod is OOM-killed; CPU over-limit → Pod is throttled (not killed).

Give the HPA desired-replica formula.

R_des = ceil(R_cur * M_cur / M_tar).

Max replicas across N nodes of capacity C with per-Pod request r?

N * floor(C / r), because Pods pack per-node and can't be split.

How does a Service know which Pods to route to?

By label selector, dynamically — Pod IPs change so it never hardcodes them.

Which K8s object runs a batch training job to completion?

A Job (or CronJob for scheduled runs); it restarts until one success.

Why use a StatefulSet for distributed training?

It gives stable network identity (worker-0, worker-1) and stable persistent volumes each rank needs.

Why set request=limit for GPU inference Pods?

GPUs can't be over-committed and memory over-commit risks OOM; matching them yields predictable Guaranteed QoS.

What prevents HPA from oscillating on tiny metric changes?

A tolerance dead-band (default ~10%) below which no scaling occurs.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho Kubernetes ek "cluster ka operating system" hai. Tum usse sirf desired state batate ho — jaise "mere model server ki 3 copies chalao, har ek ko 1 GPU aur 8 GB RAM chahiye". Uske baad Kubernetes ek loop chalata hai jise reconciliation loop kehte hain: baar-baar check karta hai ki actual state desired ke barabar hai ya nahi, aur difference ko theek karta rehta hai. Koi Pod crash ho gaya? Turant naya bana dega. Yehi thermostat wali feeling hai — tum temperature set karte ho, machine khud manage karti hai.

ML ke liye ye isliye zaroori hai kyunki inference traffic bursty hota hai aur training bade resources maangti hai. Yahan do cheezein yaad rakho: request (guaranteed minimum jo scheduler reserve karta hai — Pod ko kaunse Node pe rakhna hai ye isse decide hota hai) aur limit (maximum ceiling — memory limit cross hui to Pod OOM-killed, CPU limit cross hui to throttle). GPU/memory ke liye aksar request=limit rakhte hain kyunki GPU share nahi ho sakti.

Autoscaling ka funda simple hai. Agar 4 replicas pe CPU 90% chal raha hai aur tumhe 50% chahiye, to formula $R_{des}=\lceil R_{cur} \cdot M_{cur}/M_{tar}\rceil = \lceil 4\times 90/50\rceil = 8$ replicas. Logic: load replicas ke saath linearly bat-ta hai, to zyada replicas add karke per-replica load neeche laate hain. Chhote fluctuations pe ye scale nahi karta (10% tolerance band), taaki oscillation na ho.

Ek aur important baat — Service kabhi bhi specific Pod IP pe route nahi karta, wo labels ke through Pods dhoondta hai, kyunki Pods marte-bante rehte hain aur unke IP badalte rehte hain. Batch training ke liye Job, scheduled retraining ke liye CronJob, aur distributed training ke stable ranks ke liye StatefulSet use karo. Bas yeh mental model pakad lo: tum goal likhte ho, Kubernetes usse sach banaye rakhta hai.

WHY does ML even need Kubernetes?

The object hierarchy (build it from scratch)

Deriving resource requests: the scheduler's arithmetic

Autoscaling: deriving the HPA formula

ML-specific patterns

Common mistakes (Steel-manned)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections