Noesis
5.3.17 · D5MLOps & Deployment

Question bank — Edge deployment and ONNX

1,604 words7 min read↑ Back to topic

Before we start, three words this page leans on, in plain language:

  • Execution Provider (EP) — the actual backend (CPU, CUDA, TensorRT, CoreML, NNAPI) that does the number-crunching for an ONNX model. Think "which engine is under the hood".
  • opset — the version number of the operator dictionary. It says "these are the exact definitions of Conv, Relu, etc. that this graph was written against".
  • quantization — replacing high-precision numbers (float32, 4 bytes) with small integers (int8, 1 byte) using an affine map , where is the scale and the zero-point.

True or false — justify

Every line: decide T/F first, then check the reason.

ONNX guarantees bit-identical outputs across every runtime.
False. Different EPs use different kernels, operator fusion, and floating-point summation orders, so results agree only to a tolerance (atol≈1e-4), never bit-for-bit.
Edge deployment always beats cloud on latency.
False. Edge wins only when network + queue cost exceeds the compute penalty of the smaller chip; a heavy model on a weak device can be slower than a fast cloud GPU on a good network.
Quantizing to int8 makes the model exactly 4× smaller.
Roughly true for the weights (float32=4 bytes → int8=1 byte), but not the whole file: some layers stay float, and metadata/activations add overhead, so end-to-end shrink is a bit under 4×.
A higher opset version is always the safer choice.
False. If the target runtime doesn't support that opset, loading fails outright — pick the lowest opset that still supports your operators and matches the runtime.
ONNX stores the training loop and optimizer so you can resume training.
False. ONNX is a static inference graph — nodes, initializers (weights), typed inputs/outputs. There's no optimizer state or backprop; it describes the forward pass only.
Exporting the same model twice with the same dummy input can produce different graphs.
True in principle — if the model has data-dependent control flow (if/loops based on tensor values), tracing captures only the path the dummy took, so a differently-shaped or valued input traces a different graph.
Choosing CUDAExecutionProvider guarantees the model runs on the GPU.
False. ORT tries providers in list order and falls back to CPU for any op the GPU EP can't handle, so parts may silently run on CPU. See TensorRT and GPU Optimization for the fuller GPU path.
Edge deployment improves privacy because raw data never leaves the device.
True. Inference is local, so the sensitive input (photo, audio) is never transmitted — this is exactly the property that pairs edge with Data Privacy and Federated Learning.
The zero-point must always be .
False. only for symmetric quantization. For asymmetric ranges (e.g. ReLU outputs in ), is a nonzero integer so that real still lands on an exact integer.
Post-training quantization never hurts accuracy noticeably.
False. Outlier-sensitive layers can lose a lot; a wide range forces a large scale , and rounding error is bounded by , so the error grows. Fix with calibration or QAT. See Model Quantization and Pruning.

Spot the error

Each line states a plausible-sounding claim with a hidden bug. Find it.

"I set dynamic_axes only on the output, so batch size can vary."
Wrong axis. The input batch dimension must be declared dynamic; the output shape is derived from the input, so pinning the input to fixed batch still errors on variable batches.
"Since ONNX is standardized, I skip validating edge outputs against the original model."
Skips the tolerance check. Kernel/fusion differences shift results slightly, so you must run np.allclose(cloud_out, edge_out, atol=1e-4) before trusting the deployment.
"I quantized to int8, so I'll use a single global range for the whole tensor to keep it simple."
Loses precision. One global range is dominated by outliers, inflating and rounding error; per-channel ranges give each channel a tighter and lower error.
"My model has an if on tensor values, so I'll just trace it once and ship."
Trace freezes one branch. Tracing records only the executed path; the other branch is missing. You need torch.onnx.export with scripting (or refactor the control flow) to capture both.
"I picked opset 18 to be safe, matching my newest PyTorch."
Ignores the runtime. Safety depends on the deployment runtime's opset support, not the export framework's — a phone's older ORT may reject opset 18.
"Edge latency = inference time, so I compared only vs ."
Dropped the transfer terms. The fair cloud comparison includes ; ignoring them makes cloud look artificially fast. See Model Serving and Inference Latency.
"I dequantize with ."
Forgot the zero-point. Correct dequantization is ; dropping shifts every value by .

Why questions

Answer the reason, not just the fact.

Why does ONNX turn integration glue into ?
Each framework writes one exporter to ONNX and each runtime writes one importer from ONNX — a hub-and-spoke — instead of a custom bridge for every framework-runtime pair.
Why must the affine quant map be affine (a line), not some curve?
An affine map is the simplest order-preserving, invertible mapping, and hardware implements it with a single multiply + add — a curve would be slow and non-invertible on integer chips.
Why do we insist that real maps to an exact integer?
Padding and ReLU produce exact zeros; if zero mapped to a fractional integer, those operations would leak rounding error everywhere they touch zero.
Why does tracing an export need a real-shaped dummy input?
Export records the actual sequence of tensor ops executed — with no input there's nothing to run, so no op sequence, so no graph.
Why does a tighter quantization range reduce error?
A smaller gives a smaller scale , and the rounding error is bounded by , so shrinking shrinks the worst-case error.
Why is integer math often faster on edge CPUs than float32?
int8 weights are 1 byte vs 4, cutting memory-bandwidth traffic (usually the bottleneck), and many edge cores have fast integer/SIMD units but weak floating-point.
Why does ONNX describe the model as a graph of nodes rather than plain code?
A graph is language- and framework-neutral and lets runtimes analyse and rewrite it — fuse operators, fold constants — which raw code cannot easily expose. See Computational Graphs.
Why does QAT beat post-training quantization on hard models?
QAT simulates the rounding during training, so the weights learn to be robust to it, whereas post-training quantization bolts rounding on afterwards with no chance to adapt.

Edge cases

Degenerate and boundary scenarios — the ones that crash in production.

What is the scale if (a constant tensor)?
The numerator is , giving , which breaks dequant (). Runtimes special-case this by clamping to a tiny positive value or storing the constant directly.
If a value lies outside the calibrated , what happens?
It gets clipped to the endpoint after quantization ( saturates at or ), so extreme outliers are lost — this is why calibration range choice matters.
When is edge not worth it even with a fast network?
When the model is too large to fit device memory or the compute penalty of the small chip alone exceeds the whole cloud round-trip — then cloud wins despite the transfer cost.
What happens if the input dict key doesn't match the exported input_names?
ORT binds inputs by name; a mismatched key means the required input is unbound and session.run raises an error — the graph never executes.
If two providers both support an op, which one runs it?
ORT assigns by provider list order — the first EP in your list that supports the op claims it, so ordering ["CUDAExecutionProvider","CPUExecutionProvider"] prefers GPU with CPU fallback.
What if opset_version is set but a used operator only exists in a higher opset?
Export fails (or emits an unsupported-op error), because the operator's definition isn't available at that opset — you must raise the opset or replace the op.
On a device with no accelerator (only CPU), does CoreML/NNAPI still help?
Not for GPU/NPU speedups, but these EPs can still fuse ops and pick tuned CPU kernels; the big wins from Mobile ML - CoreML and TFLite come when a real NPU/GPU is present.
Recall One-line summary to lock it in

ONNX is a neutral static graph; runtimes agree to a tolerance not exactly; quantization trades precision for size via with error ; and edge beats cloud only when network+queue cost outweighs the small chip's compute penalty.