Edge deployment and ONNX

WHY does edge deployment exist?

Break down the four forces (the 80/20 of why edge):

Force	Cloud pain	Edge win
Latency	network RTT + queue	local compute only
Privacy	raw data leaves device	data never leaves
Cost	pay per inference call	one-time device cost
Availability	needs connectivity	works offline

Formula When is edge worth it? (a latency budget derivation)

Total response time in the cloud path: $T_{\text{cloud}} = t_{\text{up}} + t_{\text{infer}}^{\text{cloud}} + t_{\text{down}} + t_{\text{queue}}$ On edge: $T_{\text{edge}} = t_{\text{infer}}^{\text{edge}}$ Why these terms? In the cloud you must pay the two transfer terms $t_{\text{up}}, t_{\text{down}}$ (they scale with data size / bandwidth $B$ : $t \approx \text{bytes}/B$ ) plus queueing. On edge those vanish. Edge wins exactly when $T_{\text{edge}} < T_{\text{cloud}}$ , i.e. $t_{\text{infer}}^{\text{edge}} < t_{\text{up}} + t_{\text{infer}}^{\text{cloud}} + t_{\text{down}} + t_{\text{queue}}$ Equivalently, rearranging the compute terms: $\underbrace{t_{\text{infer}}^{\text{edge}} - t_{\text{infer}}^{\text{cloud}}}_{\text{compute penalty of smaller chip}} \;<\; \underbrace{t_{\text{up}} + t_{\text{down}} + t_{\text{queue}}}_{\text{network + queue cost}}$ Cleanly: edge wins when the network + queue cost exceeds the compute penalty of the smaller chip.

WHAT is ONNX?

HOW does the ONNX pipeline work?

Think of it as Train → Export → Optimize → Run.

Train in PyTorch/TF/sklearn.
Export to a .onnx graph (trace the ops).
Optimize for the target: quantization, operator fusion, constant folding.
Run with ONNX Runtime (ORT), which picks an Execution Provider (CPU, CUDA, TensorRT, CoreML, NNAPI...).

Worked example Exporting a PyTorch model (Why each step?)

import torch
dummy = torch.randn(1, 3, 224, 224)          # Why? export TRACES the graph, needs a real-shaped input
torch.onnx.export(
    model, dummy, "m.onnx",
    input_names=["img"], output_names=["logits"],
    dynamic_axes={"img": {0: "batch"}},       # Why? let batch size vary at runtime
    opset_version=17)                          # Why? opset = which operator versions the runtime must support

Why tracing needs a dummy input: ONNX records the actual sequence of tensor ops executed. No input → no trace → no graph.

Worked example Running with ONNX Runtime

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("m.onnx",
         providers=["CPUExecutionProvider"])   # Why? EP = the backend that actually computes
out = sess.run(["logits"], {"img": np.random.randn(1,3,224,224).astype("float32")})

Why the dict? ORT binds by the input_names you set at export time.

The 80/20 win: Quantization

Formula Deriving the int8 mapping from scratch

We want to represent a real value $r$ in a range $[r_{\min}, r_{\max}]$ using an 8-bit integer $q \in [q_{\min}, q_{\max}]$ (e.g. $[-128,127]$ ). Step 1 — linearity: assume an affine map $r = S(q - Z)$ . Why affine? it's the simplest invertible map preserving order, and hardware can do it with a multiply + add. Step 2 — solve for the scale $S$ by matching the endpoints: $r_{\max}-r_{\min} = S\,(q_{\max}-q_{\min}) \;\Rightarrow\; \boxed{S = \frac{r_{\max}-r_{\min}}{q_{\max}-q_{\min}}}$ Step 3 — solve for the zero-point $Z$ . We want real $0$ to map to an exact integer (padding/ReLU rely on exact zero). Set $r=0$ in the affine map: $0 = S(q_0 - Z) \;\Rightarrow\; q_0 = Z,$ so the integer representing real $0$ is exactly $Z$ . Now find $Z$ 's value using the low endpoint $r_{\min}=S(q_{\min}-Z)$ : $r_{\min}=S(q_{\min}-Z)\;\Rightarrow\; \boxed{Z = q_{\min} - \frac{r_{\min}}{S}}$ Step 4 — quantize / dequantize: $q = \text{round}\!\Big(\frac{r}{S}\Big) + Z, \qquad \hat r = S(q - Z)$ Why rounding introduces error: $|\hat r - r| \le S/2$ , so smaller $S$ (tighter range) ⇒ less error. That's why per-channel ranges beat one global range.

Worked example Numeric quantization (Why this step?)

Range $[-1, 1]$ , target int8 $[-128,127]$ .

$S = \dfrac{1-(-1)}{127-(-128)} = \dfrac{2}{255} \approx 0.007843$ . Why? apply the scale formula.
$Z = -128 - \dfrac{-1}{0.007843} = -128 + 127.5 = -0.5 \to$ round to $0$ (symmetric). Why? zero-point is the integer that represents real 0.
Quantize $r = 0.5$ : $q = \text{round}(0.5/0.007843) + 0 = \text{round}(63.75) = 64$ .
Dequantize: $\hat r = 0.007843 \times 64 = 0.502$ . Error $= 0.002 \le S/2 = 0.0039$ . ✓

Common mistakes (Steel-man + fix)

Active recall

Recall Feynman: explain to a 12-year-old

Imagine you drew a recipe (your model). Normally only your kitchen (framework) can cook it. ONNX is like translating the recipe into a language every kitchen understands — so a tiny food truck (your phone) can cook it too. And quantization is like rounding "2.37 grams of salt" to "2 grams": the dish tastes almost the same, but now you can carry way fewer measuring tools. Running it on the food truck instead of mailing ingredients to a giant restaurant across town = edge deployment: faster, private, works even with no phone signal.

Connections

Model Quantization and Pruning
Model Serving and Inference Latency
TensorRT and GPU Optimization
Computational Graphs
Data Privacy and Federated Learning
Mobile ML - CoreML and TFLite

What does ONNX stand for and what is it?

Open Neural Network Exchange — an open, framework-agnostic file format storing a model as a static computation graph (nodes, weights, typed IO).

Why does ONNX reduce integration complexity from O(N×M) to O(N+M)?

Hub-and-spoke: each framework needs one exporter TO ONNX (N) and each runtime one importer FROM ONNX (M), instead of custom glue for every pair.

What are the 4 forces favoring edge over cloud?

Lower latency, better privacy, lower per-inference cost, offline availability.

Derive the int8 scale S.

Assume affine r=S(q−Z); match endpoints: r_max−r_min = S(q_max−q_min) ⇒ S=(r_max−r_min)/(q_max−q_min).

What is the zero-point Z and why does it matter?

Z = q_min − r_min/S; it is the integer that real 0 maps to (set r=0 ⇒ q0=Z), critical for exact ReLU/padding.

Max error introduced by quantization rounding?

|r̂ − r| ≤ S/2, so smaller scale (tighter range) means less error.

Why does torch.onnx.export need a dummy input?

Export TRACES the executed ops; without a real-shaped input there is no execution to record into a graph.

What is an Execution Provider in ONNX Runtime?

The backend that actually computes the ops (CPU, CUDA, TensorRT, CoreML, NNAPI...); ORT dispatches graph nodes to it.

Why can't you assume bit-identical outputs across runtimes?

Different EPs use different kernels/fusion/FP ordering; results agree only within a tolerance, so validate with np.allclose.

What is opset_version and its failure mode?

The operator-set version the model requires; if the target runtime doesn't support it, loading fails. Use the lowest opset that covers your ops.

QAT vs post-training quantization — why prefer QAT?

QAT simulates rounding during training so weights adapt to quantization noise, preserving accuracy better than post-hoc PTQ.

What is dynamic_axes used for?

Declaring dimensions (e.g. batch) that can vary at runtime, avoiding shape-mismatch errors in production.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, edge deployment ka matlab hai model ko wahin chalana jahan data banta hai — phone, camera, ya sensor par — cloud ke door server par bhejne ke bajaye. Iska fayda? Cloud path me har prediction ke liye data upload karo, wait karo, phir answer download karo — yeh latency, bandwidth cost, aur privacy teeno kharab karta hai, aur agar network down ho toh sab thap. Edge par yeh transfer waale terms zero ho jaate hain, isliye chhota chip thoda slow hone ke bawajood overall fast aur reliable nikalta hai. Formal condition: edge tab jeetta hai jab chhote chip ka compute penalty (t_infer_edge − t_infer_cloud) network+queue cost (t_up + t_down + t_queue) se kam ho.

Ab ONNX ek "sabki samajh me aane waali file format" hai — model ko ek static computation graph (nodes = operators jaise Conv/MatMul, plus weights, plus typed inputs/outputs) ke roop me store karta hai. Isse framework (PyTorch) aur runtime (phone ka NNAPI) decouple ho jaate hain. Bina ONNX ke aapko har framework × runtime pair ke liye alag glue likhna padta (N×M kaam). ONNX hub-and-spoke bana deta hai: har framework ek exporter, har runtime ek importer — total N+M kaam. Pipeline yaad rakho: Train → Export → Optimize → Run.

Sabse bada 80/20 trick hai quantization: float32 weight 4 byte ka hota hai, use int8 (1 byte) me convert karo toh model 4x chhota aur memory-bandwidth bottleneck kam. Formula khud derive hota hai: maan lo affine map $r = S(q-Z)$ , endpoints match karke $S = (r_{max}-r_{min})/(q_{max}-q_{min})$ . Zero-point $Z$ nikalne ke liye real 0 ko affine map me daalo (r=0 ⇒ q0=Z), yani $Z$ hi wo integer hai jo real 0 ko represent karta hai — ReLU/padding ke liye zaroori. Rounding error zyada se zyada $S/2$ hota hai.

Common galtiyan: (1) "export karo toh har jagah bilkul same output" — nahi, alag Execution Providers ke kernels alag hote hain, sirf tolerance ke andar match hota hai, np.allclose se check karo. (2) "quantization free accuracy" — nahi, calibration data ya QAT use karo. (3) "opset jitna naya utna accha" — nahi, runtime jo support kare wahi opset lo. Bas yeh samajh lo toh interview aur production dono clear.

WHY does edge deployment exist?

WHAT is ONNX?

HOW does the ONNX pipeline work?

The 80/20 win: Quantization

Common mistakes (Steel-man + fix)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections