Exercises — Edge deployment and ONNX

(a) Edge — data device se bahar na jaana privacy solve karta hai. (b) Cloud — ek chhota edge chip 40 GB model hold nahi kar sakta; sirf cloud GPUs kar sakte hain. (c) Edge — 5 ms bina network ke matlab round-trip terms $t_{\text{up}},t_{\text{down}}$ exist hi nahi karte; locally compute karna padega.

nodes (operators jaise Conv, MatMul, Relu), initializers (the weights), typed inputs, aur typed outputs. Parent se mnemonic: N-O-R-I = Nodes, Operators, Runtime-Inputs.

Ek EP woh backend hai jo actually graph ke operations compute karta hai — woh swappable engine (CPU, CUDA, TensorRT, CoreML, NNAPI) jise ONNX Runtime har operator deta hai. Related: TensorRT and GPU Optimization, Mobile ML - CoreML and TFLite.

float32 = 4 bytes/weight: $50\times 10^6\times 4=200\times 10^6$ bytes $=200$ MB. int8 = 1 byte/weight: $50\times 10^6\times 1=50\times 10^6$ bytes $=50$ MB. Exactly 4× shrink — parent ka "80/20 win".

$S=\frac{r_{\max }-r_{\min }}{q_{\max }-q_{\min }}=\frac{6-(-4)}{127-(-128)}=\frac{10}{255}\approx 0.039216$ Yeh formula kyun: yeh 10-wide real range ko 255-wide integer range mein fit karne ke liye stretch karta hai taaki dono endpoints line up ho jaayein.

$Z=q_{\min }-\frac{r_{\min }}{S}=-128-\frac{-4}{0.039216}=-128+102=-26$ Kyun: $Z$ woh integer hai jo exactly real $0$ tak decode hota hai. Kyunki range $[-4,6]$ asymmetric hai, $Z$ zero nahi hai — yeh crowded side ki taraf slide karta hai.

$q=\text{round}\left(\frac{2.5}{0.039216}\right)+(-26)=\text{round}(63.75)-26=64-26=38$ $\hat{r}=S(q-Z)=0.039216(38-(-26))=0.039216\times 64=2.510$ Error $=|2.510-2.5|=0.010\le S/2=0.0196$. ✓ theoretical bound ke andar hai.

Scheme A: $S_A=\frac{1-(-1)}{255}=\frac{2}{255}\approx 0.007843$, toh $S_A/2\approx 0.003922$. Scheme B: $S_B=\frac{0.1-(-0.1)}{255}=\frac{0.2}{255}\approx 0.000784$, toh $S_B/2\approx 0.000392$. Per-channel error 10× chhoti hai ($0.000392$ vs $0.003922$). Kyun: busy channel ki true values sirf $\pm 0.1$ span karti hain; global range apne 255 integer steps ka zyaadaatar empty space $[0.1,1]$ par waste karta hai, jisse har step coarse ho jaata hai. Tighter range ⇒ chhota $S$ ⇒ chhota error. Dekho Model Quantization and Pruning.

$T_{\text{cloud}}=30+8+12+20=70\text{ ms},T_{\text{edge}}=55\text{ ms}$ Edge jeetta hai $70-55=15$ ms se. Parent ki rearranged inequality check karo — compute penalty $t_{\text{infer}}^{\text{edge}}-t_{\text{infer}}^{\text{cloud}}=55-8=47$ ms; network+queue $=30+12+20=62$ ms. Kyunki $47<62$, edge jeetta hai. ✓ Dekho Model Serving and Inference Latency.

$T_{\text{edge}}=95$ ms vs $T_{\text{cloud}}=70$ ms. Cloud ab jeetta hai $25$ ms se. Compute penalty $95-8=87$ ms ab network+queue $62$ ms se zyaada ho gayi, isliye inequality flip ho gayi. Lesson: winner "edge vs cloud" ki property nahi hai — yeh numbers par depend karta hai — hamesha inequality evaluate karo.

torch.onnx.export(model, dummy, "m.onnx",
    input_names=["img"], output_names=["logits"],
    dynamic_axes={"img": {0: "batch"}},  # variable batch → axis 0 dynamic mark karo
    opset_version=16)                     # lowest opset jo woh op support karta hai

• dummy (real-shaped tensor): export actual ops ko trace karta hai; koi input nahi ⇒ koi trace nahi ⇒ koi graph nahi.
• dynamic_axes: iske bina, batch dummy ke size par freeze ho jaata hai aur production feeds shape-mismatch errors dete hain.
• opset_version=16: lowest opset choose karo jo phir bhi tumhara op support kare, taaki runtimes ka widest set ise load kar sake. Zaroorat se zyaada high opset "unsupported opset" load failures ka risk deta hai.

$3\times 10^{-5}=0.00003<1\times 10^{-4}=0.0001$, isliye allclose True return karta hai — yeh pass hota hai. Tumhe bit-identical output force karne ki koshish nahi karni chahiye: alag EPs alag kernels, operator fusion, aur floating-point summation order use karte hain, isliye results sirf ek tolerance tak agree karte hain, exactly kabhi nahi. Agreement validate karna (equality nahi) sahi engineering check hai.

Transfer time $\approx D/B$ (bytes over bytes-per-second). Edge par deploy karo jab: $t_{\text{infer}}^{\text{edge}}-t_{\text{infer}}^{\text{cloud}}<\frac{D}{B}+t_q$ Left = chhote chip ki compute penalty; right = network + queue cost jo bachti hai. Plug in: penalty $=0.05-0.008=0.042$ s. Network+queue $=\frac{1.5\times 10^6}{5\times 10^6}+0.02=0.3+0.02=0.32$ s. Kyunki $0.042<0.32$ → edge par deploy karo (har call mein $0.32-0.042=0.278$ s bachta hai). Data Privacy and Federated Learning se ties karta hai jab privacy bhi round-trip rok deti hai.

$S=10/255\approx 0.039216$, $Z=-26$ (Q2.2–Q2.3 se). $q=\text{round}\left(\frac{-3.2}{0.039216}\right)+(-26)=\text{round}(-81.6)-26=-82-26=-108$ Clamp check: $-108\in [-128,127]$ ✓ (koi saturation nahi). $\hat{r}=S(q-Z)=0.039216(-108-(-26))=0.039216\times (-82)=-3.2157$ Error $=|-3.2157-(-3.2)|=0.0157\le S/2=0.0196$. ✓ Round-trip guaranteed error box ke andar land karta hai.

$q=\text{round}(9.0/0.039216)+(-26)=\text{round}(229.5)-26=230-26=204$. Lekin int8 max $127$ hai, isliye $204$ ko clamp (saturate) karke 127 karna padega. Dequantized: $\hat{r}=0.039216(127-(-26))=0.039216\times 153=6.0$ — true $9.0$ crush hokar $6.0$ ban jaata hai, $3.0$ ka bada error. Kyun: $9.0$ calibrated range $[-4,6]$ se bahar hai. Fix: range ko ek calibration dataset se choose karo jo aisi outlier values actually cover kare, ya quantization-aware training (QAT) use karo taaki network out-of-range activations se bachna seekhe. Dekho Model Quantization and Pruning.