Exercises — Edge deployment and ONNX

(a) Edge — data never leaves the device solves privacy. (b) Cloud — a tiny edge chip cannot hold a 40 GB model; only cloud GPUs can. (c) Edge — 5 ms with no network means the round-trip terms $t_{\text{up}},t_{\text{down}}$ don't even exist; must compute locally.

nodes (operators like Conv, MatMul, Relu), initializers (the weights), typed inputs, and typed outputs. Mnemonic from the parent: N-O-R-I = Nodes, Operators, Runtime-Inputs.

An EP is the backend that actually computes the graph's operations — the swappable engine (CPU, CUDA, TensorRT, CoreML, NNAPI) that ONNX Runtime hands each operator to. Related: TensorRT and GPU Optimization, Mobile ML - CoreML and TFLite.

float32 = 4 bytes/weight: $50\times 10^6\times 4=200\times 10^6$ bytes $=200$ MB. int8 = 1 byte/weight: $50\times 10^6\times 1=50\times 10^6$ bytes $=50$ MB. Exactly a 4× shrink — the "80/20 win" from the parent.

$S=\frac{r_{\max }-r_{\min }}{q_{\max }-q_{\min }}=\frac{6-(-4)}{127-(-128)}=\frac{10}{255}\approx 0.039216$ Why this formula: it stretches the 10-wide real range to fit the 255-wide integer range so both endpoints line up.

$Z=q_{\min }-\frac{r_{\min }}{S}=-128-\frac{-4}{0.039216}=-128+102=-26$ Why: $Z$ is the integer that decodes back to exactly real $0$. Because the range $[-4,6]$ is asymmetric, $Z$ is not $0$ — it slides toward the crowded side.

$q=\text{round}\left(\frac{2.5}{0.039216}\right)+(-26)=\text{round}(63.75)-26=64-26=38$ $\hat{r}=S(q-Z)=0.039216(38-(-26))=0.039216\times 64=2.510$ Error $=|2.510-2.5|=0.010\le S/2=0.0196$. ✓ within the theoretical bound.

Scheme A: $S_A=\frac{1-(-1)}{255}=\frac{2}{255}\approx 0.007843$, so $S_A/2\approx 0.003922$. Scheme B: $S_B=\frac{0.1-(-0.1)}{255}=\frac{0.2}{255}\approx 0.000784$, so $S_B/2\approx 0.000392$. Per-channel error is 10× smaller ($0.000392$ vs $0.003922$). Why: the busy channel's true values only span $\pm 0.1$; a global range wastes most of its 255 integer steps on empty space $[0.1,1]$, making each step coarse. Tighter range ⇒ smaller $S$ ⇒ smaller error. See Model Quantization and Pruning.

$T_{\text{cloud}}=30+8+12+20=70\text{ ms},T_{\text{edge}}=55\text{ ms}$ Edge wins by $70-55=15$ ms. Check the parent's rearranged inequality — compute penalty $t_{\text{infer}}^{\text{edge}}-t_{\text{infer}}^{\text{cloud}}=55-8=47$ ms; network+queue $=30+12+20=62$ ms. Since $47<62$, edge wins. ✓ See Model Serving and Inference Latency.

$T_{\text{edge}}=95$ ms vs $T_{\text{cloud}}=70$ ms. Cloud now wins by $25$ ms. The compute penalty $95-8=87$ ms now exceeds network+queue $62$ ms, so the inequality flips. Lesson: the winner is not a property of "edge vs cloud" but of the numbers — always evaluate the inequality.

torch.onnx.export(model, dummy, "m.onnx",
    input_names=["img"], output_names=["logits"],
    dynamic_axes={"img": {0: "batch"}},  # variable batch → mark axis 0 dynamic
    opset_version=16)                     # lowest opset that has the op

• dummy (real-shaped tensor): export traces the actual ops; no input ⇒ no trace ⇒ no graph.
• dynamic_axes: without it, batch is frozen at the dummy's size and production feeds cause shape-mismatch errors.
• opset_version=16: pick the lowest opset that still supports your op, so the widest set of runtimes can load it. Higher-than-needed opset risks "unsupported opset" load failures.

$3\times 10^{-5}=0.00003<1\times 10^{-4}=0.0001$, so allclose returns True — it passes. You should not try to force bit-identical output: different EPs use different kernels, operator fusion, and floating-point summation order, so results agree only to a tolerance, never exactly. Validating agreement (not equality) is the correct engineering check.

Transfer time $\approx D/B$ (bytes over bytes-per-second). Deploy on edge when: $t_{\text{infer}}^{\text{edge}}-t_{\text{infer}}^{\text{cloud}}<\frac{D}{B}+t_q$ Left = compute penalty of the smaller chip; right = network + queue cost saved. Plug in: penalty $=0.05-0.008=0.042$ s. Network+queue $=\frac{1.5\times 10^6}{5\times 10^6}+0.02=0.3+0.02=0.32$ s. Since $0.042<0.32$ → deploy on edge (saves $0.32-0.042=0.278$ s per call). Ties to Data Privacy and Federated Learning when privacy also forbids the round-trip.

$S=10/255\approx 0.039216$, $Z=-26$ (from Q2.2–Q2.3). $q=\text{round}\left(\frac{-3.2}{0.039216}\right)+(-26)=\text{round}(-81.6)-26=-82-26=-108$ Clamp check: $-108\in [-128,127]$ ✓ (no saturation). $\hat{r}=S(q-Z)=0.039216(-108-(-26))=0.039216\times (-82)=-3.2157$ Error $=|-3.2157-(-3.2)|=0.0157\le S/2=0.0196$. ✓ Round-trip lands inside the guaranteed error box.

$q=\text{round}(9.0/0.039216)+(-26)=\text{round}(229.5)-26=230-26=204$. But int8 max is $127$, so $204$ must clamp (saturate) to 127. Dequantized: $\hat{r}=0.039216(127-(-26))=0.039216\times 153=6.0$ — the true $9.0$ is crushed to $6.0$, a huge error of $3.0$. Why: $9.0$ lies outside the calibrated range $[-4,6]$. Fix: choose the range with a calibration dataset that actually covers such outliers, or use quantization-aware training (QAT) so the network learns to avoid out-of-range activations. See Model Quantization and Pruning.