ONNX Runtime and Portable Specialist Models

A portable model is not a model that merely has an .onnx suffix.

Portability means that the graph’s operator domains and versions are understood by the target runtime, every custom operator is present, preprocessing and label order travel with the weights, the chosen execution provider actually claims the intended graph, and the output stays within an accepted numerical and product-quality envelope. A file can be valid ONNX and still fail every one of those conditions.

This chapter builds a complete small-model boundary in Rust. The companion generates a real ONNX protobuf containing a MatMul → Add → Softmax classifier, binds its digest to a verified manifest, loads it with ONNX Runtime through the ort crate, verifies the runtime-discovered tensor schema, executes two inputs, validates the returned probabilities, and emits release receipts. Corruption, oversized input, manifest drift, unsupported accelerators, inconsistent thread configuration, bad feature domains, and malformed graphs are tested as first-class cases.

The classifier is intentionally tiny and synthetic. Its purpose is to make the deployment mechanics observable, not to claim useful evidence-quality predictions. Replace its three-node graph with an evaluated embedding model, reranker, detector, OCR classifier, VAD, safety head, or other specialist only after the same boundary survives.

Learning objectives

By the end you will be able to:

explain the relationship among an ONNX model, IR version, operator domain, opset import, graph, nodes, initializers, value information, and external tensor data;
distinguish syntactic model validity from semantic, numerical, and product validation;
decide when a portable specialist is better than a general generative model;
export a graph with explicit input/output names, types, shapes, dynamic axes, preprocessing, and label order;
pin the Rust binding, ONNX Runtime API level, native runtime build, model revision, and model digest independently;
understand why native runtime acquisition is part of the software supply chain;
configure a session with explicit execution providers, graph optimization, threading, memory pattern, and allocator policy;
detect rather than hide CPU fallback;
validate runtime-discovered inputs and outputs before serving traffic;
create a Rust tensor without changing layout, dtype, or ownership accidentally;
validate probability vectors beyond “the call returned Ok”;
reason about execution-provider graph partitioning and host/device transfer boundaries;
choose static versus dynamic shapes and understand their optimization consequences;
compare dynamic, static, QOperator, and QDQ quantization;
separate session startup, warm-up, preprocessing, transfer, kernel, synchronization, and postprocessing latency;
know when I/O binding matters and when it only adds complexity;
operate multiple small models without multiplying thread pools and arenas accidentally;
treat an untrusted model as a resource-exhaustion and parser attack surface;
design parity tests, numerical tolerance gates, performance gates, and release rollback for a specialist model.

Prerequisites and dependency order

Complete Candle fundamentals first. Candle exposed storage, dtype, shape, layout, device, parameter names, and graph execution directly. Complete mistral.rs local serving to understand a separate generative service boundary. This chapter covers a different deployment shape: an application-embedded, task-specific graph with a narrow typed input and output.

Review model artifacts for immutable bundle identity, quantization and device placement for capacity accounting, and provider-neutral capability traits before exposing this implementation to the rest of an application.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example onnx_specialist_model

The program creates and loads a real 337-byte ONNX model. Its identity is:

model ID       mosaic/evidence-disposition-linear-v1
model SHA-256  958051d6f8abf1a11e7aa8320c1eacdecac957a4dbe364efac1d72261c4693e0
IR version     8
opset          ai.onnx 13
input          features      float32 [1,4]
output         probabilities float32 [1,3]
labels         accept, review, reject

The runtime receipt identifies:

Rust binding       ort 2.0.0-rc.13
ORT API minor      27
native build       rel-1.28.0, commit da9b5e3
requested EP       CPU
selected EP        CPUExecutionProvider
CPU fallback       false
intra-op threads   1
inter-op threads   1
parallel graph     false
optimization       basic

The first evidence input normalizes to [0.9, 0.8, 0.75, 0.1] and produces:

accept  0.9408883
review  0.058663875
reject  0.00044790408

The second input normalizes to [0.1, 0.2, 0.0, 0.9] and predicts reject. Both results bind the model digest and the same runtime identity into distinct inference-receipt digests.

Run the focused tests:

cargo test -p mosaic-ml --all-features onnx_runtime
cargo test -p mosaic-ml --all-features --example onnx_specialist_model

The implementation is in crates/mosaic-ml/src/onnx_runtime.rs; the complete executable is crates/mosaic-ml/examples/onnx_specialist_model.rs.

Why specialist models belong beside generative models

A generative model can describe an image, decide whether speech is present, rank passages, or classify a document. That does not make it the best component for each job.

A specialist often has:

a narrower input contract;
a bounded output space;
smaller resident weights;
lower and more predictable latency;
less output parsing;
easier calibration;
simpler offline evaluation;
greater edge portability;
no prompt-injection surface when its input is numerical;
a failure that can be expressed as a typed unavailable capability.

Examples include:

Capability	Typical input	Typical output
embedding	token IDs and mask	fixed-width vector
reranker	query/document tokens	relevance score
image classifier	normalized image tensor	class probabilities
object detector	image tensor	boxes, labels, scores
OCR recognizer	cropped line tensor	token sequence
voice activity detector	audio window	speech probability
audio event classifier	log-mel frames	event probabilities
safety head	embedding or hidden state	policy labels
quality estimator	engineered or learned features	quality score

The correct architecture is rarely “replace every foundation-model call.” Use specialists where a stable task definition and adequate labeled data exist. Keep a larger model for ambiguous synthesis, open-world reasoning, or escalation. The harness can combine them: a VAD trims silence, a detector finds candidate regions, a reranker narrows evidence, and a generative model explains the retained material with citations.

ONNX from first principles

ONNX defines a portable serialized computation graph. The top-level ModelProto associates metadata and version declarations with a GraphProto. The graph contains:

named inputs and outputs;
computation nodes;
constant initializers such as trained weights;
optional value/type/shape information;
optional local functions and sparse values.

A node names an operator, a domain, ordered inputs, ordered outputs, and attributes. The built-in operator set lives in the default domain. Vendor or project extensions use other domains. Every model imports a (domain, opset version) pair for each operator set it needs.

Three versions must not be conflated:

ONNX IR version governs the serialized model structure.
Operator-set version governs operator schemas and semantics.
ONNX Runtime version/API level governs what a particular runtime build can parse and run.

An operator’s stable schema does not change in place. New semantics appear at a new operator version. The model imports an opset; the runtime resolves each node against that import. This is why copying “opset 13” from one exporter command without understanding its operators is not a portability plan.

The companion uses IR 8 and the default-domain opset 13. Its nodes are:

features [1,4]
  │
  ├── MatMul(weight [4,3]) → weighted [1,3]
  ├── Add(bias [3])        → logits [1,3]
  └── Softmax              → probabilities [1,3]

The official operator documentation is the source of truth for input ranks, broadcasting, attributes, type constraints, and version differences. Do not infer those rules from one framework’s eager operation.

Initializers and external data

Small weights can live in TensorProto.raw_data. Large models may use external data, where tensor bytes live outside the main protobuf. External data improves handling of very large files, but it changes the artifact from one file into a bundle.

For an external-data release, bind at least:

the main model digest;
every external file’s relative path, byte count, and digest;
a total byte and file-count limit;
a rule rejecting absolute paths, .., symlinks escaping the bundle, and remote locations;
the exact resolved bundle-root identity.

Hashing only model.onnx while permitting mutable weights.bin is not model pinning.

Static, symbolic, and runtime shapes

An ONNX graph may declare a concrete dimension, a symbolic dimension, or an unknown dimension. Shape inference can propagate some types and shapes through a graph, but it cannot solve every data-dependent computation. Static declarations are not runtime proof, and inferred shapes are not application limits.

The application must still cap runtime batch, sequence, image, and feature dimensions before allocating a tensor. For a fixed specialist, prefer static dimensions when they match the product: they make schema drift obvious and enable stronger optimization. Use dynamic axes when the actual workload requires them, then define independent minima, maxima, bucketing, and memory admission.

The export contract

Export is a release pipeline, not a developer convenience command. A complete export record should contain:

source framework and exporter versions;
source checkpoint revision and digest;
export code revision;
training/evaluation dataset lineage;
model purpose and prohibited uses;
input names, dtypes, ranks, static/symbolic dimensions, and bounds;
preprocessing version and test vectors;
output names, dtypes, shapes, label order, and postprocessing;
IR version and every opset import;
custom operator libraries and their digests;
external-data inventory;
numerical parity results against the source runtime;
target execution providers and tested hardware;
latency, memory, binary-size, and quality results;
license and redistribution terms.

Prefer an export function checked into the repository over a notebook cell copied from history. Run the ONNX checker and shape inference during export. Inspect the graph with a viewer, but do not mistake visual inspection for automated validation.

Parity is task-aware

Bitwise equality across runtimes and devices is often unrealistic. “Close enough” without a defined metric is equally useless.

Choose parity criteria by output:

logits: absolute/relative error plus top-k agreement;
probabilities: finite/range/sum checks, distribution distance, class agreement, and calibration;
embeddings: norm bounds, cosine similarity, and retrieval-set agreement;
boxes: coordinate tolerance, IoU, class agreement, and NMS parity;
sequences: token equality or task metric, including termination behavior;
audio: waveform/spectrogram distance plus perceptual/task evaluation.

Evaluate boundary inputs, not only representative averages. Include zero-like inputs, maximum allowed shapes, unusual Unicode or media aspect ratios upstream, class ties, low-confidence outputs, and preprocessing values exactly on quantization or normalization boundaries.

Generating the chapter model without Python

The function tiny_specialist_onnx_model writes a deterministic protobuf using small, private wire-format helpers. It emits:

ModelProto.ir_version = 8;
producer name and version;
one default-domain opset import at version 13;
three nodes;
weight and bias float32 initializers;
exact input and output value information.

Hand-encoding protobuf is appropriate here because the graph is 337 bytes and the exercise exposes what an exporter creates. It is not a recommendation to build a general ONNX exporter by hand. A production exporter should use maintained framework/ONNX libraries, validation, and canonical test fixtures.

The model computes:

logits = features × weight + bias
probabilities = softmax(logits)

Its weights reward confidence, extraction coverage, and independent agreement; they penalize anomaly for accept and reward it for reject. Those weights are pedagogical constants, not a trained policy. The model’s limitations are explicit so nobody can accidentally market its output as evidence trust.

Pinning the Rust and native runtime

The workspace uses:

ort = {
  version = "=2.0.0-rc.13",
  default-features = false,
  features = [
    "api-27",
    "copy-dylibs",
    "download-binaries",
    "std",
    "tls-rustls",
  ],
}

The exact pre-release pin matters because the ort 2.0 API is still release-candidate software. Version 2.0.0-rc.13 targets ONNX Runtime 1.28 and can expose API level 27. The companion records both the binding version and runtime build information instead of assuming they are identical.

The download-binaries feature is convenient for this learning workspace: its native runtime is downloaded during dependency build and copied for execution. It is also a supply-chain decision. The ort release process publishes artifact digests and build attestations. A production organization may instead:

build ONNX Runtime from a pinned Microsoft source revision;
verify a published artifact and SLSA attestation;
mirror an approved binary internally;
use load-dynamic with a digest-verified absolute library path;
ship a platform package produced by its own release pipeline.

Do not allow an arbitrary environment variable or current working directory to select a native library in a privileged service. Record the library bytes, source, compile flags, enabled execution providers, and dependent accelerator libraries as release inputs.

The example reports:

ORT Build Info: git-branch=rel-1.28.0,
git-commit-id=da9b5e3,
fp8-kv-cache=1,
build type=Release

That string is evidence about the loaded build. It is not a substitute for a verified binary digest in a hardened deployment.

Loading in the correct order

PortableEvidenceClassifier::load uses a fail-closed sequence:

Validate the manifest structure.
Validate runtime-policy bounds and consistency.
Reject an empty or oversized model before asking a parser to inspect it.
Compute SHA-256 and require the exact manifest digest.
Resolve the requested execution provider or return a typed error.
Create a session builder.
Register the explicit CPU provider and allocator policy.
Apply optimization, thread, execution-mode, and memory-pattern settings.
Commit the verified bytes into an ONNX Runtime session.
Inspect the runtime-discovered inputs and outputs.
Require one exact float32 [1,4] input named features.
Require one exact float32 [1,3] output named probabilities.
Build model and runtime receipts and hash them into a runtime identity.

The digest check occurs before native model parsing. A digest allowlist does not make the parser invulnerable, but it drastically narrows which bytes reach it.

Why validate the schema twice?

The manifest declares what the release is supposed to contain. The session reports what the runtime actually loaded. Requiring both catches:

the right file paired with the wrong manifest;
an exporter changing an input name;
a label head changing width;
float16 replacing float32;
a formerly static dimension becoming dynamic;
an unexpected second input such as attention_mask;
an output being reordered or renamed.

Calling the first model input by position and accepting the first output by position makes these changes silent. Name and schema validation converts them into release failures.

Preprocessing is part of the model

The model receives four normalized values. The public application input stores integers:

pub struct EvidenceFeatures {
    pub source_confidence_bps: u16,
    pub extraction_coverage_bps: u16,
    pub independent_agreements: u8,
    pub anomaly_bps: u16,
}

Basis points avoid accepting NaN or infinity at the domain boundary. Normalization checks:

confidence ≤ 10,000
coverage   ≤ 10,000
agreements ≤ 4
anomaly    ≤ 10,000

Then it maps to float32:

confidence / 10,000
coverage   / 10,000
agreements / 4
anomaly    / 10,000

For images, the equivalent contract includes decode limits, orientation, color space, resize geometry, interpolation, crop/letterbox policy, channel order, tensor layout, scale, mean, standard deviation, and dtype. For audio it includes decode, channel mix, sample rate, amplitude normalization, window, hop, feature transform, and padding. For text it includes normalization, tokenizer files, special tokens, truncation side, padding side, and mask semantics.

Changing any one of those can produce shape-valid nonsense. Give preprocessing its own version, golden fixtures, and digest. Ideally include it in the model bundle manifest and inference receipt.

Creating tensors and validating results

The checked features become one owned tensor:

let input = Tensor::<f32>::from_array((
    [1usize, 4],
    normalized.to_vec().into_boxed_slice(),
))?;

let outputs = session.run(ort::inputs!["features" => input])?;
let (shape, probabilities) =
    outputs["probabilities"].try_extract_tensor::<f32>()?;

The production code does not stop there. It requires:

exact output shape [1,3];
exactly three values;
every value finite;
every value in the closed interval [0,1];
the sum within 1e-4 of one.

Then it chooses the greatest probability with a deterministic tie rule and maps the index through the manifest’s exact label list. The inference receipt binds raw features, normalized features, probabilities, selected label, model digest, and runtime identity.

Probability validation does not prove calibration. A vector can be mathematically valid and systematically overconfident. Calibration and decision thresholds belong to the evaluated release, not to the ONNX parser.

Execution providers and the fallback trap

ONNX Runtime uses execution providers to assign supported nodes or subgraphs to CPU, CUDA, TensorRT, Core ML, OpenVINO, QNN, XNNPACK, and other backends. According to its architecture, providers are considered in priority order and the default CPU provider completes nodes not claimed elsewhere.

That makes heterogeneous execution possible. It also creates an observability trap:

requested CUDA
  ├── most graph claimed by CUDA
  ├── unsupported island runs on CPU
  ├── tensors cross device boundaries twice
  └── request succeeds, latency regresses

“CUDA provider registered” does not mean “the complete graph ran on CUDA.” Release acceptance should inspect provider assignment/profiling and decide whether partial fallback is allowed.

This learning crate compiles only CPU. Asking for CUDA or Core ML fails with BackendNotCompiled unless the caller explicitly enables recorded CPU fallback. In the fallback case the receipt preserves both:

requested_provider = cuda
selected_provider  = CPUExecutionProvider
used_cpu_fallback  = true

In production, create separate build features and acceptance jobs for each supported EP. Do not put accelerator initialization behind “best effort” in a latency-sensitive service.

Provider parity

For every release and EP combination, run:

schema and load tests;
golden numerical parity;
task-quality evaluation;
boundary-shape cases;
warm and cold latency distributions;
peak host and device memory;
concurrency and cancellation;
complete provider-assignment inspection;
restart and rollback.

An ONNX model is portable only across the combinations you actually verify.

Graph optimization and portability

ONNX Runtime applies graph transformations before and after provider partitioning. Basic optimizations are intended to be semantics-preserving. Extended and layout optimizations may fuse or rewrite graphs in provider- and hardware-sensitive ways.

The fixture uses Basic so its teaching graph remains easy to reason about. A production CPU release may benefit from All, but benchmark and validate the result. When saving an optimized model offline, use the exact target options and hardware. A graph optimized for one provider or instruction set may not preserve the portability of the original ONNX artifact.

Separate these identities:

source ONNX model digest;
optimizer tool/runtime version;
optimization configuration;
optimized artifact digest;
target provider and hardware capability;
numerical and quality acceptance report.

The ORT format can reduce startup or enable smaller runtime builds. It is a deployment artifact, not a replacement for the portable source model. Keep the source ONNX graph and conversion recipe so the release can be reproduced and inspected.

Threading, sessions, and concurrency

ONNX Runtime separates:

intra-op parallelism: work inside an operator;
inter-op parallelism: independent graph nodes executing concurrently;
request concurrency: multiple application inferences;
session concurrency: multiple sessions and their allocators/thread pools.

The official threading guide notes that CPU defaults often use physical cores and that parallel graph execution can hurt models without useful branches. Adding eight application workers, each with an eight-thread session, does not create eight-way parallelism; it can create 64 runnable threads plus runtime and server work.

The fixture chooses:

intra-op = 1
inter-op = 1
sequential graph execution

That is not a universal performance recommendation. It makes the release deterministic and avoids oversubscription in tests. Tune with the complete service workload.

The current ort Rust API takes &mut Session for run because some runtime internals are not safe for concurrent access through the same wrapper. Reasonable designs include:

one dedicated blocking worker owning one session;
a bounded pool of workers/sessions;
dynamic batching before one session;
model-specific service processes for stronger isolation.

Never hold an async reactor thread inside a long native inference call. Admit with a semaphore, send work to a bounded blocking or dedicated pool, propagate deadlines and cancellation state, and reconcile resources even if native execution cannot be preempted immediately.

Thread pools and arenas multiply

Each session can own thread pools and an arena allocator. Loading ten tiny models naïvely may cost more in runtime structures and reserved memory than in weights. ONNX Runtime’s C API supports global thread pools, shared allocators, shared initializers, and prepacked-weight containers for some deployments. Measure process resident memory after warm-up, not just model file sizes.

An arena can retain peak memory. One unusual dynamic-shape request may grow it permanently unless the deployment uses an appropriate shrink or isolation policy. Static shapes and request limits reduce that uncertainty.

I/O binding and device transfers

For CPU specialists, normal input/output tensors are often sufficient. For accelerator EPs, host copies can dominate a small graph. ONNX Runtime’s I/O binding guide explains how to bind inputs and outputs in target-device memory.

I/O binding matters when:

an upstream stage already produced a device tensor;
multiple graphs form a device-resident pipeline;
output shape is known and buffers can be reused;
profiling shows host/device copies are material.

It does not automatically improve a tiny one-off request whose data begins and ends on CPU. Device-resident buffers also add lifetime, allocator, synchronization, and cancellation concerns.

Measure these phases separately:

decode/preprocess
host allocation
host → device
kernel queue
kernel execution
device synchronization
device → host
postprocess

A timer around run() can hide or misattribute asynchronous device work. Force the synchronization required by the product outcome before reporting completed latency.

Quantization without magical thinking

Quantization changes storage, kernels, calibration requirements, accuracy, and sometimes graph shape. ONNX Runtime documents dynamic and static quantization:

dynamic quantization computes activation parameters during inference;
static quantization uses representative calibration data before deployment;
QOperator graphs use quantized operator forms;
QDQ graphs surround values with quantize/dequantize nodes and can map well to some EPs.

Do not infer speed from a smaller file. An EP may lack the intended quantized kernel, insert conversions, or fall back to CPU. Older GPUs may gain nothing from INT8. A poorly representative calibration set can preserve aggregate accuracy while damaging rare but important slices.

A quantized release needs a new:

model digest and format inventory;
calibration-dataset identity;
quantization configuration;
provider/hardware matrix;
task-quality and slice report;
probability calibration report;
latency, throughput, memory, and power report;
rollback target.

For embeddings and rerankers, test downstream ranking behavior rather than only tensor error. For detection, test box and class metrics. For a safety classifier, false-negative slices matter more than mean absolute logit error.

Performance method for portable specialists

Benchmark the product boundary, then decompose it.

Cold path

Measure:

native library load;
model read and digest verification;
protobuf parse;
graph validation and optimization;
provider discovery/initialization;
weight allocation and prepacking;
accelerator compilation/cache creation;
first inference.

Warm path

Measure:

preprocessing;
admission wait;
tensor creation;
transfers;
inference;
postprocessing;
response serialization.

Report distributions, not one duration: p50, p95, p99, maximum, sample count, warm-up count, batch, shape, provider, hardware, power mode, thread policy, concurrent load, model digest, and runtime identity.

Profile before tuning. ONNX Runtime can emit detailed operator/thread timing in a trace viewable with Perfetto-compatible tools; see the profiling guide. Confirm whether time is in preprocessing, copies, one operator, provider compilation, allocator churn, or oversubscription.

Batching

Batching improves throughput only when:

the model declares or is exported for the batch dimension;
the EP kernels exploit the larger work;
padding and collation do not dominate;
the waiting window fits the latency SLO;
a large request cannot monopolize admission.

Use bounded batch items and bytes. Preserve an index mapping from each output row to the exact input receipt. Never sort or drop inputs without carrying identity.

Security: a model is untrusted structured input

ONNX Runtime explicitly warns that a malicious model can consume excessive memory or compute even when it conforms to the specification. Treat externally sourced models like untrusted native-input documents.

Defenses include:

accept only verified model and external-data digests;
cap files, bytes, dimensions, tensor elements, nodes, initializers, and custom metadata during an independent inspection stage;
reject unknown operator domains and custom operators by default;
resolve external data inside a sealed bundle root;
load and test new models in a sandbox with CPU, memory, file, network, and time limits;
deny network access during model load;
run as an unprivileged identity with a read-only filesystem;
separate model approval from runtime activation;
use canary traffic and a bounded rollback;
keep runtime and native dependencies patched;
fuzz the application’s manifest and preprocessing parsers;
do not deserialize executable framework checkpoints when a data-only format is available.

Custom operators are native code. A signed model digest says nothing about an unpinned custom-op shared library. Bind its digest and ABI, review it as code, and isolate it accordingly.

A hard inference deadline does not guarantee native compute stops at that instant. Capacity accounting must distinguish client abandonment, task cancellation requested, native work still running, and resources actually released.

Observability and release receipts

At model load, record:

integration release;
binding and native runtime identity;
model and bundle digests;
IR and opset imports;
schema;
preprocessing version;
requested and selected providers;
provider options;
optimization level;
thread and allocator policy;
load duration and warm-up result.

Per inference, record bounded metadata:

request and model release IDs;
input shape/dtype and preprocessing receipt;
queue, preprocessing, inference, and postprocessing duration;
selected provider or service instance;
output shape and validation result;
decision label/threshold version;
cancellation/timeout state;
error class.

Do not log raw media, tokens, embeddings, or complete probability vectors by default. They may be sensitive and high-cardinality. Use trace sampling and explicitly governed diagnostic captures.

The companion’s runtime identity is the SHA-256 of its runtime and model receipts. Each inference receipt then binds that identity to raw and normalized input plus output. This gives replay a specific release target rather than “whatever model is currently installed.”

Failure investigations

Model loads on a laptop but not on an edge target

Check, in order:

exact model digest and external-data inventory;
IR and opset compatibility;
custom/contrib operator domains;
reduced runtime’s operator and type configuration;
execution-provider library availability;
static/dynamic shape support;
native ABI and dependent library versions.

Do not “fix” it by exporting with an arbitrary older opset until parity tests show what changed.

Accelerator deployment is slower than CPU

Collect an operator/provider profile. Look for a small graph, repeated host/device copies, CPU fallback islands, provider compilation on the measured path, tiny batches, unsupported layouts, or oversubscribed host preprocessing. Compare synchronized end-to-end latency with the same pre/postprocessing.

Outputs are valid probabilities but labels are wrong

Compare label order, preprocessing version, channel/layout order, normalization constants, and source-runtime golden vectors. A softmax cannot tell you that index 0 was renamed from cat to dog.

Latency variance grows after adding more models

Inventory sessions, per-session thread pools, arenas, prepacked weights, and request worker pools. Test shared resources or fewer dedicated workers. Check NUMA placement on large hosts. One model’s parallel execution can contend with another model’s preprocessing.

Quantized model is smaller but quality regresses

Compare float and quantized intermediate outputs, calibration coverage, per-class/slice metrics, decision thresholds, and EP assignment. Do not tune thresholds on the held-out test set merely to hide a quantization regression.

Memory stays high after one large request

Confirm whether dynamic dimensions admitted the request, whether the arena retained its peak, whether output buffers remain referenced, and whether an accelerator cache expanded. Fix admission and isolation before adding process restarts as the primary control.

Testing and release gates

The chapter companion provides twelve module tests and one complete-example test:

stable, sub-kilobyte real model generation;
actual ONNX Runtime inference;
probability range and sum validation;
distinct trusted/suspicious decisions;
digest mismatch before runtime loading;
malformed manifest rejection;
byte limit before parser/allocator;
malformed protobuf reaching a typed runtime error;
typed unavailable accelerator;
explicit, recorded CPU fallback;
invalid and internally inconsistent thread policy;
feature-domain validation before inference;
inference receipts changing with one basis point of input;
model/runtime identity shared across complete-example results.

A production model needs additional gates:

exporter reproducibility;
source-runtime versus ONNX parity;
task-quality and calibration evaluation;
every supported EP/hardware combination;
dynamic/boundary/adversarial inputs;
cold/warm latency and memory;
concurrency, backpressure, timeout, cancellation, restart, and rollback;
dependency/license/advisory inspection;
model-card and prohibited-use review;
canary analysis against the previous release.

Keep model release independent from application release when operationally useful, but never allow the application to load an unrecognized mutable “latest” model.

Alternatives and trade-offs

Option	Strength	Cost
Candle-native model	Rust tensor/model control and one language	you own architecture parity and backend coverage
ONNX Runtime via `ort`	broad exporter and EP ecosystem; narrow embedded API	native runtime supply chain and provider-specific behavior
mistral.rs service	high-level local generative and multimodal serving	process/protocol operations and larger model footprint
provider API	managed models and elastic capacity	data boundary, network, provider drift, cost
task-specific native library	mature optimized implementation	another ABI and domain-specific integration

ONNX is especially attractive when training/export happens in Python but a Rust application needs a portable, narrow inference surface. It does not eliminate the need to understand the exported graph.

Exercises

1. Manifest contract

Add a preprocessing_sha256 and label_schema_version to VerifiedOnnxModel. Make them mandatory, validate their syntax, and include them in both model and inference identity.

2. Independent oracle

Implement the fixture’s matrix multiplication, bias, and stable softmax in plain Rust. Compare all three ONNX probabilities with a tolerance of 1e-6 over at least 100 generated valid inputs.

3. Contract corruption

Generate four graph variants: renamed input, float64 input, output width four, and a second required input. Prove each fails at session-contract validation rather than during ordinary inference.

4. Batch export

Change the fixture to [N,4] → [N,3] with a symbolic batch dimension. Add application bounds 1..=32, preserve input-to-output identity, and test batches 1, 2, 32, 0, and 33.

5. Provider policy

Design a configuration in which CUDA may fall back for only a named allowlist of operators. Specify how provider assignment is collected and what fails release acceptance.

6. Quantization experiment

Export or obtain a small float and INT8 classifier. Compare file size, cold load, p50/p95/p99, peak memory, class agreement, calibration, and per-slice quality on CPU. Explain any case where INT8 loses.

7. Worker topology

Design a Tokio service for four specialist models under a 16-core budget. Set bounded queues, session/worker counts, intra-op threads, deadlines, and cancellation accounting. Explain how the design prevents oversubscription.

8. Untrusted-model lab

Create a sandbox acceptance process for a newly submitted ONNX bundle. Define filesystem, network, CPU, memory, time, operator-domain, external-data, and promotion controls. Include rollback.

Solution guidance

1. Manifest contract

Use exact 64-character lowercase hex for the preprocessing digest and a nonempty version with a tight length/character grammar. A manifest that changes either must produce a new runtime identity. An inference replay should fail before tensor creation if the required preprocessing implementation is unavailable.

2. Independent oracle

Use the same [4,3] row-major weights. Compute logits with checked finite arithmetic. Subtract the maximum logit before exp, divide by the finite positive sum, then compare component-wise. The oracle must not call ONNX Runtime or reuse its output.

3. Contract corruption

Keep each mutated model’s digest consistent with its manifest so the load reaches the schema gate. If the manifest itself reflects the mutation, use an immutable application release contract separate from the supplied manifest. The lesson is that a self-asserted manifest alone cannot authorize a changed schema.

4. Batch export

Encode a symbolic dimension in the TensorShapeProto, but enforce concrete runtime bounds before allocation. Collect outputs by row and zip them with immutable request IDs. Reject zero and 33 before Tensor::from_array.

5. Provider policy

Capture the optimized graph or provider profile in an acceptance job. Compare every assigned node against the expected provider and allowlist. Fail on a new CPU island, extra transfer boundary, or latency regression even if output parity passes.

6. Quantization experiment

Warm both releases identically and use the same inputs and thread policy. Include preprocessing in end-to-end results but also report inference-only timing. Slice by class and hard examples. Quantization can lose on a tiny model when conversion overhead or absent kernels outweigh reduced memory bandwidth.

7. Worker topology

Start with one dedicated session worker per model and one intra-op thread, then increase only from measurements. Give each model a bounded queue and the whole service a global admission limit. Expired queued work should be removed before inference; abandoned native work remains charged until the worker reports completion.

8. Untrusted-model lab

Use a disposable, unprivileged process/container with a read-only bundle, no network, explicit resource limits, and a timeout enforced outside the process. First inspect structure and domains, then load, run golden/adversarial inputs, collect resource profiles, and sign a promotion manifest. Production activates only promoted digests and retains the previous approved release.

Check your understanding

Why are IR version, opset version, and runtime version separate?
Why is a valid ONNX protobuf not necessarily safe or useful?
What does execution-provider registration prove, and what does it not prove?
Why can partial CPU fallback make an accelerator slower?
Why is label order part of model identity?
When are static shapes preferable?
What does I/O binding remove from the hot path?
Why can more session threads reduce service throughput?
What extra artifacts must be hashed for external tensor data?
Why does an INT8 file need a new quality and calibration report?

Answers: the three versions govern different contracts; syntax cannot prove resource behavior, semantics, or quality; registration proves availability/configuration, not complete graph assignment; fallback adds transfers and fragmented execution; probabilities are positional; static shapes strengthen validation and optimization when product bounds are fixed; I/O binding can avoid implicit host/device allocation and copies; nested parallelism oversubscribes cores and increases contention; hash every external file plus its safe relative path and size; quantization changes numerics, kernels, provider assignment, and decision behavior.

ONNX Runtime and Portable Specialist Models

Learning objectives

Prerequisites and dependency order

Completion artifact

Why specialist models belong beside generative models

ONNX from first principles

Initializers and external data

Static, symbolic, and runtime shapes

The export contract

Parity is task-aware

Generating the chapter model without Python

Pinning the Rust and native runtime

Loading in the correct order

Why validate the schema twice?

Preprocessing is part of the model

Creating tensors and validating results

Execution providers and the fallback trap

Provider parity

Graph optimization and portability

Threading, sessions, and concurrency

Thread pools and arenas multiply

I/O binding and device transfers

Quantization without magical thinking

Performance method for portable specialists

Cold path

Warm path

Batching

Security: a model is untrusted structured input

Observability and release receipts

Failure investigations

Model loads on a laptop but not on an edge target

Accelerator deployment is slower than CPU

Outputs are valid probabilities but labels are wrong

Latency variance grows after adding more models

Quantized model is smaller but quality regresses

Memory stays high after one large request

Testing and release gates

Alternatives and trade-offs

Exercises

1. Manifest contract

2. Independent oracle

3. Contract corruption

4. Batch export

5. Provider policy

6. Quantization experiment

7. Worker topology

8. Untrusted-model lab

Solution guidance

1. Manifest contract

2. Independent oracle

3. Contract corruption

4. Batch export

5. Provider policy

6. Quantization experiment

7. Worker topology

8. Untrusted-model lab

Check your understanding

Completion checklist

Primary references