Edge deployment and ONNX

5.3.17 · AI-ML › MLOps & Deployment

Edge deployment kyun exist karta hai?

Chaar forces ko samjhein (yahi hai why edge ka 80/20):

Force	Cloud ka dard	Edge ka faida
Latency	network RTT + queue	sirf local compute
Privacy	raw data device chhodta hai	data kabhi nahi jaata
Cost	har inference call par pay	ek baar device cost
Availability	connectivity chahiye	offline kaam karta hai

ONNX pipeline kaise kaam karta hai?

Ise Train → Export → Optimize → Run samjhein.

Train PyTorch/TF/sklearn mein.
Export karein ek .onnx graph mein (ops ko trace karein).
Optimize karein target ke liye: quantization, operator fusion, constant folding.
Run karein ONNX Runtime (ORT) se, jo ek Execution Provider choose karta hai (CPU, CUDA, TensorRT, CoreML, NNAPI...).

Worked example PyTorch model export karna (har step kyun?)

import torch
dummy = torch.randn(1, 3, 224, 224)          # Why? export graph ko TRACE karta hai, real-shaped input chahiye
torch.onnx.export(
    model, dummy, "m.onnx",
    input_names=["img"], output_names=["logits"],
    dynamic_axes={"img": {0: "batch"}},       # Why? runtime par batch size vary karne do
    opset_version=17)                          # Why? opset = runtime ko kaunse operator versions support karne chahiye

Tracing ko dummy input kyun chahiye: ONNX actually execute hue tensor ops ka actual sequence record karta hai. Koi input nahi → koi trace nahi → koi graph nahi.

Worked example ONNX Runtime se run karna

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("m.onnx",
         providers=["CPUExecutionProvider"])   # Why? EP = woh backend jo actually compute karta hai
out = sess.run(["logits"], {"img": np.random.randn(1,3,224,224).astype("float32")})

Dict kyun? ORT un input_names se bind karta hai jo aapne export time par set kiye the.

80/20 win: Quantization

Formula Scratch se int8 mapping derive karna

Hum chahte hain ki ek real value $r$ jo $[r_{m i n}, r_{m a x}]$ range mein hai, use ek 8-bit integer $q \in [q_{m i n}, q_{m a x}]$ (jaise $[- 128, 127]$ ) se represent karein. Step 1 — linearity: ek affine map $r = S (q - Z)$ assume karein. Affine kyun? yeh sabse simple invertible map hai jo order preserve karta hai, aur hardware ise multiply + add se kar sakta hai. Step 2 — scale $S$ solve karein endpoints match karke: $r_{m a x} - r_{m i n} = S (q_{m a x} - q_{m i n}) \Rightarrow S = \frac{r _{m a x} - r _{m i n}}{q _{m a x} - q _{m i n}}$ Step 3 — zero-point $Z$ solve karein. Hum chahte hain ki real $0$ ek exact integer se map ho (padding/ReLU exact zero par rely karte hain). Affine map mein $r = 0$ set karein: $0 = S (q_{0} - Z) \Rightarrow q_{0} = Z,$ to real $0$ ko represent karne wala integer exactly $Z$ hai. Ab low endpoint $r_{m i n} = S (q_{m i n} - Z)$ use karke $Z$ ki value dhundhein: $r_{m i n} = S (q_{m i n} - Z) \Rightarrow Z = q_{m i n} - \frac{r _{m i n}}{S}$ Step 4 — quantize / dequantize: $q = round (\frac{r}{S}) + Z, \overset{r}{^} = S (q - Z)$ Rounding error kyun introduce karta hai: $∣ \overset{r}{^} - r ∣ \leq S /2$ , to chhhota $S$ (tighter range) ⇒ kam error. Isliye per-channel ranges ek global range se better hote hain.

Common mistakes (Steel-man + fix)

Active recall

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tumne ek recipe banai (tumhara model). Normally sirf tumhari kitchen (framework) ise cook kar sakti hai. ONNX ek aisi language mein recipe translate karna hai jo har kitchen samjhe — taaki ek chhhoti food truck (tumhara phone) bhi ise cook kar sake. Aur quantization yeh hai jaise "2.37 gram namak" ko "2 gram" kar dena: dish almost waisi hi lagti hai, par ab tumhe bahut kam measuring tools saath laane padte hain. Ise shahar ke doosri taraf ke bade restaurant mein ingredients mail karne ki jagah food truck par chalana = edge deployment: faster, private, bina phone signal ke bhi kaam karta hai.

Connections

Model Quantization and Pruning
Model Serving and Inference Latency
TensorRT and GPU Optimization
Computational Graphs
Data Privacy and Federated Learning
Mobile ML - CoreML and TFLite

ONNX ka full form kya hai aur yeh kya hai?

Open Neural Network Exchange — ek open, framework-agnostic file format jo model ko ek static computation graph (nodes, weights, typed IO) ke roop mein store karta hai.

ONNX integration complexity ko O(N×M) se O(N+M) par kyun laata hai?

Hub-and-spoke: har framework ko sirf ek exporter TO ONNX chahiye (N) aur har runtime ko sirf ek importer FROM ONNX (M), bajaaye har pair ke liye custom glue ke.

Edge ko cloud se behtar banane wale 4 forces kaun se hain?

Kam latency, better privacy, kam per-inference cost, offline availability.

Int8 scale S derive karo.

Affine r=S(q−Z) assume karo; endpoints match karo: r_max−r_min = S(q_max−q_min) ⇒ S=(r_max−r_min)/(q_max−q_min).

Zero-point Z kya hai aur yeh kyun matter karta hai?

Z = q_min − r_min/S; yeh woh integer hai jis par real 0 map hota hai (r=0 set karo ⇒ q0=Z), exact ReLU/padding ke liye zaroori hai.

Quantization rounding se introduce hone wali max error?

|r̂ − r| ≤ S/2, to chhhota scale (tighter range) matlab kam error.

torch.onnx.export ko dummy input kyun chahiye?

Export execute hue ops ko TRACE karta hai; real-shaped input ke bina koi execution nahi, koi graph nahi record hoga.

ONNX Runtime mein Execution Provider kya hota hai?

Woh backend jo actually ops compute karta hai (CPU, CUDA, TensorRT, CoreML, NNAPI...); ORT graph nodes ko usi par dispatch karta hai.

Alag runtimes mein bit-identical outputs kyun nahi man sakte?

Alag EPs alag kernels/fusion/FP ordering use karte hain; results sirf ek tolerance ke andar agree karte hain, isliye np.allclose se validate karo.

opset_version kya hai aur iska failure mode kya hai?

Model jo operator-set version maangta hai; agar target runtime ise support nahi karta, to loading fail hoti hai. Apne ops cover karne wala lowest opset use karo.

QAT vs post-training quantization — QAT kyun prefer karein?

QAT training ke dauran rounding simulate karta hai taaki weights quantization noise ke saath adapt ho sakein, PTQ se zyada accuracy preserve karta hai.

dynamic_axes kis kaam aata hai?

Woh dimensions declare karne ke liye (jaise batch) jo runtime par vary kar sakti hain, production mein shape-mismatch errors se bachne ke liye.