Candle Fundamentals and Tensor/Model Integration

A tensor program can execute successfully and still be the wrong model.

The file may contain a projection trained for another tokenizer. Every matrix multiplication may be shape-valid while a weight was transposed relative to the exporting framework. A Metal request may silently run on CPU. An f16 conversion may overflow a reduction. A safe checkpoint format may contain a billion-element tensor declaration, non-finite values, or extra parameters that the application never intended to accept. A benchmark may measure only accelerator command submission rather than completed work. None of those failures necessarily produces a Rust panic or a Candle error.

This chapter therefore treats Candle as a tensor engine inside a larger model contract. We will load a real safetensors file, validate its bytes and every parameter, build two candle_nn::Linear layers, execute a batch, normalize the outputs, and return a receipt that identifies the checkpoint, device, shapes, dtype, and integration release. We will also make a transposed, non-contiguous tensor visible rather than letting layout remain an invisible performance detail.

The worked artifact is intentionally a tiny deterministic encoder, not a pretrained foundation model. Its small size lets us calculate every output, corrupt each boundary in tests, and learn the mechanics before mistral.rs, ONNX Runtime, diffusion, speech, and video engines add large graphs and backend-specific behavior.

Learning objectives

By the end you will be able to:

explain a tensor in terms of storage, dtype, shape, strides/layout, device, and graph history;
distinguish a view operation from a materializing operation;
reason about broadcasting, reductions, matrix multiplication, and Linear weight layout;
identify when a non-contiguous layout is valid but costly or unsupported by a later kernel;
choose CPU, CUDA, or Metal deliberately and report any fallback;
separate checkpoint storage dtype, parameter dtype, activation dtype, accumulation dtype, and output dtype;
parse a real safetensors checkpoint without confusing “no arbitrary code” with “trusted model”;
enforce checkpoint byte, tensor-count, element-count, name, shape, dtype, and finiteness limits;
understand how VarBuilder prefixes map model module paths to checkpoint parameter names;
construct and test an exact Candle inference graph;
validate outputs independently of whether the tensor operations returned Ok;
connect a local Candle implementation to a provider-neutral embedding capability;
isolate blocking tensor work from an async request reactor;
reason about batching, residency, memory peaks, warm-up, and truthful latency measurement;
distinguish inference from training and know when gradient state is actually needed;
diagnose CPU/GPU disagreements without changing several variables at once;
define production acceptance gates for a new model and backend combination.

Prerequisites and dependency order

Read transformer mechanics for embeddings, attention, normalization, and logits; model artifacts for bundle identity; quantization, memory, batching, and device placement for capacity math; and provider-neutral capability traits for the application boundary.

This chapter follows multimodal retrieval because a local embedding or reranking model is one way to populate that funnel. It precedes the engine-specific chapters because Candle teaches the primitive tensor and checkpoint contracts that higher-level servers often hide.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example candle_model_pipeline

The example creates a real 436-byte safetensors checkpoint in memory. It contains four f32 parameters and 23 scalar values:

Parameter	Shape	Values	Meaning
`encoder.weight`	`[3, 4]`	12	four input features to three hidden features
`encoder.bias`	`[3]`	3	encoder bias
`projection.weight`	`[2, 3]`	6	three hidden features to two output features
`projection.bias`	`[2]`	2	projection bias

The exact checkpoint SHA-256 is:

948510e79a91d4427da34c36a7dc3237a8b63855d178070118f90a533be8dd1c

Two input rows flow through:

input [2,4]
  → Linear(4,3)
  → ReLU
  → Linear(3,2)
  → row-wise L2 normalization
  → unit embeddings [2,2]

The resulting embeddings are:

[ 0.94449675, -0.32852063]
[ 0.83205026,  0.55470020]

The report also transposes a [2,3] tensor. The [3,2] result is a non-contiguous view until .contiguous() materializes:

[[1, 4],
 [2, 5],
 [3, 6]]

The output records requested = cpu, selected = cpu, and used_cpu_fallback = false. That device receipt is part of the model result rather than an unobservable log line.

What Candle is—and what it is not

Candle is a Rust machine-learning framework from Hugging Face. Its project is divided into layers:

candle-core supplies tensors, dtypes, devices, operations, layouts, and storage;
candle-nn supplies neural-network modules, variable builders, optimizers, and common layers;
candle-transformers contains model implementations and transformer utilities;
repository examples demonstrate model families and applications;
separate integrations can load formats or delegate graphs to other runtimes.

This companion directly depends only on candle-core and candle-nn for this milestone. It does not copy a full transformer implementation, download weights, or claim that a two-layer fixture represents production embedding quality. Later chapters compare higher-level local serving and portable specialist-model paths.

Candle gives the program tensor operations and backend implementations. It does not infer:

which model release a file belongs to;
whether a tokenizer and preprocessing recipe match those weights;
whether extra or missing parameters should be accepted;
whether CPU fallback satisfies a product SLO;
whether an embedding is normalized as an index expects;
whether a model output is semantically correct;
whether an untrusted model artifact is permitted for this tenant;
whether a batch should be admitted under current memory pressure.

Those remain application responsibilities.

A tensor has more than values

At minimum, reason about a tensor as:

Tensor = (storage, dtype, shape, layout, device, graph history)

Storage and dtype

Storage is the buffer holding scalar representations. Dtype says how to interpret those bits: u8, i64, f16, bf16, f32, and so on. Two tensors with the same shape but different dtypes have different memory sizes, numerical ranges, precision, kernels, and often different backend support.

Storage dtype and compute dtype are separate decisions. Quantized weights might be stored in four-bit groups, dequantized into f16 blocks, accumulated in f32, and emitted as f32. Calling all of that “an f16 model” is too imprecise for a reproducible system.

For N elements, an uncompressed tensor’s payload is approximately:

bytes = N × bytes_per_element

That is not the process peak. Add temporary casts, workspace buffers, activations, allocator fragmentation, backend state, and concurrent batches. A conversion can briefly require both old and new storage.

Shape

Shape assigns meaning to axes. [2,4] in the example is two rows with four features each. Shape equality is necessary but not always sufficient: [batch, sequence, hidden] and [sequence, batch, hidden] may contain the same number of values and still encode different semantics.

Use named variables and validate dimensions at system boundaries:

let input = Tensor::from_slice(
    &flattened,
    (batch_size, input_dimension),
    &device,
)?;

Avoid accepting arbitrary dynamic shapes merely because a runtime can. Bound every dimension and checked-multiply element counts before allocating.

Layout and strides

Shape tells how many logical coordinates exist. Layout tells how those coordinates address storage. A contiguous row-major [2,3] matrix:

[[1, 2, 3],
 [4, 5, 6]]

can be transposed to shape [3,2] by changing how indices map to the same storage. No six values need to be copied. That view is logically:

[[1, 4],
 [2, 5],
 [3, 6]]

but adjacent logical values are not adjacent under the new layout. The companion proves:

let transposed = source.t()?;
assert!(!transposed.is_contiguous());
let materialized = transposed.contiguous()?;
assert!(materialized.is_contiguous());

A view is valuable because it avoids a copy. A later kernel may nevertheless require contiguous input or run more slowly with that stride pattern. .contiguous() is therefore not a cosmetic call: it is a potentially large allocation and memory transfer. Record it in performance analysis.

Reshape deserves the same care. A reshape can remain a view only when the current layout permits the requested interpretation. Slicing can introduce offsets and non-unit strides. Broadcasting may represent repeated logical values without repeated storage. Never infer allocation cost from the output shape alone.

Device

The device identifies where storage and kernels live. A CPU tensor and a CUDA tensor with the same values are not interchangeable without a transfer. Device movement has latency, memory, and synchronization consequences.

Keep model parameters and inputs on the same intended device. Repeatedly copying each request’s weights or intermediate tensors between CPU and accelerator can dominate computation. Stable production services usually load one model instance per worker/device and move only bounded inputs and outputs across the boundary.

Graph history

In training, operations may retain information needed for reverse-mode differentiation. In inference, retaining training-only state wastes memory. Candle’s differentiation model is explicit: trainable Var values participate in gradient computation and backward produces a gradient store. It is not necessary to emulate a global PyTorch-style no_grad switch around this immutable VarBuilder::from_tensors inference example. The important design rule is to construct inference from immutable tensors and avoid creating trainable variables or calling backward unless the training path requires them.

Operations from first principles

Elementwise operations

relu, addition, multiplication, square, and square root act elementwise when shapes match or the broadcasting rules permit it. ReLU is:

ReLU(x) = max(0, x)

It changes values but not shape. The example’s negative hidden activations become zero.

Broadcasting

Broadcasting expands a compatible dimension logically rather than necessarily copying data. The example computes one norm per row with shape [batch,1] and divides an output of shape [batch,2]:

let norms = logits
    .sqr()?
    .sum_keepdim(1)?
    .sqrt()?;
let normalized = logits.broadcast_div(&norms)?;

Keeping the reduced dimension produces [batch,1], which can broadcast across the output-feature axis. Removing that dimension would yield [batch]; depending on the operation’s alignment rules, that may not mean “one scalar per row.” Write the expected shapes beside reductions.

Broadcasting errors can also be silent semantic errors. A bias [hidden] should broadcast across batch and sequence axes. A mistaken [batch] tensor may accidentally broadcast along a feature axis when dimensions coincide. Tests must include unequal dimensions that expose such mistakes.

Reductions and numerical stability

Sum, mean, maximum, and norm reduce one or more axes. Always name:

the reduced axes;
whether dimensions are retained;
the accumulation dtype;
behavior for empty input;
tolerance or failure behavior for non-finite results.

The L2 norm is:

||x||₂ = sqrt(Σᵢ xᵢ²)

and normalization is:

x̂ = x / ||x||₂

The fixture’s nonzero biases ensure its two worked rows have nonzero norms, but the generic model would need an explicit zero-norm policy. Adding an epsilon silently changes the output contract; rejecting the row makes failure explicit. Pick and test one policy rather than relying on division behavior.

In reduced precision, squaring large values can overflow and adding many small values can lose precision. Many production kernels accumulate selected reductions in f32 even when activations are f16 or bf16. Confirm the exact backend kernel rather than assuming.

Matrix multiplication and `Linear`

For input X shaped [batch, in], Candle’s Linear stores weight W as [out, in] and computes the equivalent of:

Y = X Wᵀ + b

so Y has shape [batch,out]. This storage convention matches common exported checkpoints but is easy to reverse when hand-authoring weights. The fixture makes it explicit:

Tensor::from_slice(
    &[
        1.0, 0.0, 0.0, 0.0, // hidden output 0
        0.0, 1.0, 0.0, 0.0, // hidden output 1
        0.0, 0.0, 1.0, 1.0, // hidden output 2
    ],
    (3, 4),
    &device,
)

The comments describe output rows, not input columns. A shape check catches [4,3], but it cannot catch a wrongly transposed square matrix. Golden-vector tests are indispensable.

For the first input [1,2,3,4], the hidden vector after the first layer and ReLU is:

h = [1, 2, 7]

The second layer and bias produce:

z₀ = 1 + 0.5×7 + 0.1 = 4.6
z₁ = 2 - 0.5×7 - 0.1 = -1.6

Its norm is approximately 4.8693, producing:

[4.6, -1.6] / ||[4.6, -1.6]||₂
  = [0.94449675, -0.32852063]

That hand calculation proves more than “shape [1,2] came out.”

Device selection is a deployment contract

Candle supports CPU and optional accelerator backends such as CUDA and Metal, subject to crate features, host platform, compiler/toolchain, driver, and hardware availability. The installation guide documents backend-specific setup. “Candle supports CUDA” does not mean every binary was compiled with CUDA.

The companion’s learning binary is deliberately CPU-only:

pub enum DeviceRequest {
    Cpu,
    Cuda { ordinal: usize },
    Metal { ordinal: usize },
}

An accelerator request has two permitted outcomes:

return typed BackendNotCompiled; or
use CPU only when the caller explicitly allows fallback, and record that fallback.

It never silently turns Cuda { ordinal: 1 } into CPU. Silent fallback can preserve correctness while destroying latency, cost, and concurrency assumptions. It can also hide a broken production image for weeks.

Do not make one “all backends” feature

It is tempting to declare local cuda and metal features and rely on:

cargo test --all-features

But Metal is platform-specific and CUDA needs a compatible toolkit. Enabling both in a universal CI job turns optional deployment targets into mandatory build dependencies. Prefer a matrix:

portable job: CPU features, Linux/macOS
CUDA job:     CUDA feature, pinned Linux image, driver/toolkit compatibility checks
Metal job:    Metal feature, pinned macOS runner and supported architecture

Each artifact should expose a build capability receipt. Runtime selection then intersects:

requested backend
∩ compiled backends
∩ detected hardware/runtime
∩ model/dtype kernel support
∩ operational policy

No intersection means a typed admission failure unless the request explicitly permits a recorded fallback.

Accelerator timing and synchronization

Accelerator operations can enqueue work asynchronously. Timing only the enqueue path reports command-submission latency, not inference latency. Warm the exact graph, synchronize the device at the measurement boundary using the backend-supported mechanism, and time end-to-end transfer and completion separately from kernel-only work. CPU execution in this fixture is synchronous, but that fact must not be generalized to every backend.

Dtype is a chain of decisions

Use a dtype receipt with at least:

checkpoint storage dtype
loaded parameter dtype
activation dtype
accumulation/reduction dtype
output dtype
quantization scheme and group metadata, if any

`f32`

f32 is the fixture’s portable baseline. It has broad operator support and enough precision for the hand-computed golden values. It consumes twice the parameter and activation memory of f16 or bf16.

`f16`

f16 provides a smaller exponent range and more mantissa precision than bf16. It can be fast on supported accelerators, but reductions, exponentials, normalization, and large activations need care. CPU operator support and performance vary.

`bf16`

bf16 retains the exponent range of f32 with fewer mantissa bits. It often tolerates large magnitudes better than f16, but fine-grained precision is lower. Again, availability depends on device and kernel.

Conversion is not free

A checkpoint conversion can:

allocate a second full parameter copy;
round values;
change golden outputs;
choose a slow unsupported kernel path;
force later implicit casts;
invalidate a previously calibrated quantizer or embedding index.

Convert once during controlled load or artifact preparation, verify the output tolerance, and record both source and selected dtype. Avoid per-request whole-model casts.

Safetensors is a format, not a trust decision

The safetensors format starts with an eight-byte little-endian JSON-header length, followed by the JSON header and packed tensor bytes. Entries describe dtype, shape, and byte offsets. The format avoids executable pickle payloads and supports bounded metadata inspection.

That is a substantial security improvement. It does not establish that:

the file came from the expected publisher;
the digest matches a reviewed model release;
declared tensors fit local memory;
tensor names match this architecture;
tensor values are finite;
the model license permits the intended use;
the model behavior is safe;
the file has not been swapped between verification and mapping.

The format permits empty tensors and floating-point NaN/Infinity representations. Application validation still matters.

Preflight before parsing or allocation

The companion checks the byte limit before calling the loader:

if bytes.is_empty() || bytes.len() > policy.max_checkpoint_bytes {
    return Err(CandleRuntimeError::CheckpointTooLarge { ... });
}

A production loader should also enforce:

trusted or quarantined acquisition path;
exact artifact digest and optional signature/attestation;
maximum header length;
maximum tensor count;
maximum rank and per-dimension size;
checked element-count and byte-range arithmetic;
non-overlapping, in-file data offsets;
total parameter and expected resident-memory budgets;
accepted dtypes;
exact or versioned parameter-name schema;
model/license/policy admission;
no mutable artifact after verification.

Never allocate from shape.iter().product() without checked multiplication.

Buffered loading versus memory mapping

The example uses:

candle_core::safetensors::load_buffer(bytes, &device)

It already owns a bounded byte buffer and avoids an unsafe mapping boundary. Candle also exposes memory-mapped safetensors builders for large files. Mapping avoids reading the entire file into an additional user buffer and can improve startup/residency behavior, but the relevant API is unsafe because the program must uphold file-lifetime and mutation invariants.

For memory mapping:

open a stable, immutable artifact by a safe directory/file-descriptor policy;
verify identity without a path-swap race;
reject writable or externally mutable backing files;
retain the mapping for every tensor view that depends on it;
handle truncation and I/O faults;
be cautious on network filesystems;
separate virtual address space from resident memory in metrics.

“Zero-copy” does not mean zero page faults, zero device transfer, or zero memory pressure.

Exact parameter contracts with `VarBuilder`

VarBuilder connects module paths to named tensors. The companion creates a builder over a validated tensor map:

let builder = VarBuilder::from_tensors(tensors, DType::F32, &device);
let encoder = candle_nn::linear(4, 3, builder.pp("encoder"))?;
let projection = candle_nn::linear(3, 2, builder.pp("projection"))?;

builder.pp("encoder") makes the layer request encoder.weight and encoder.bias. builder.pp("projection") requests the corresponding projection names. Deeper modules compose prefixes.

A convenient builder does not prove that the complete checkpoint is correct. Some loaders only request parameters reached by construction; unrelated extra tensors can go unnoticed. Before building, the wrapper compares exact sets:

actual names == {
  encoder.bias,
  encoder.weight,
  projection.bias,
  projection.weight
}

It then validates each shape, requires f32, checks every value is finite, uses checked arithmetic for total values, and records a deterministic receipt.

Why reject extra names? An extra tensor may mean:

a checkpoint for a different architecture;
an exporter naming mismatch;
an adapter that was expected to be active but is ignored;
an accidental merge of shards;
malicious or wasteful payload;
a legitimate optional head under a different declared schema.

If optional parameters are valid, encode that in a versioned parameter schema. Do not replace exactness with “ignore whatever we do not use.”

The complete load boundary

The constructor order is deliberate:

validate model dimensions and policy
  → reject empty/oversized bytes
  → resolve requested device
  → parse bounded safetensors
  → validate exact names/shapes/dtypes/counts/finiteness
  → hash and record checkpoint
  → construct modules from validated tensors

The order limits work and makes failures attributable. Selecting a device before parsing ensures tensors are created where intended; verifying the byte bound first prevents an oversized input from reaching the parser. The digest binds the original file bytes, not a reserialized tensor map.

The receipt includes:

pub struct CheckpointReceipt {
    pub checkpoint_sha256: String,
    pub checkpoint_size_bytes: usize,
    pub format: String,
    pub parameters: Vec<ParameterReceipt>,
    pub total_parameter_values: usize,
}

For a production model, extend it with model-bundle identity, source URI without credentials, license decision, signature/attestation, tokenizer/preprocessor digest, architecture release, quantization configuration, load duration, selected device, and backend/library versions.

The forward boundary

The public method accepts rows only after validating:

non-empty batch;
batch size no greater than eight in the fixture policy;
every row exactly four values;
every input finite.

It then executes:

let hidden = self.encoder.forward(&input)?.relu()?;
let logits = self.projection.forward(&hidden)?;
let norms = logits.sqr()?.sum_keepdim(1)?.sqrt()?;
let normalized = logits.broadcast_div(&norms)?;

After Candle returns success, the wrapper independently requires:

hidden shape = [batch, 3]
output shape = [batch, 2]
output dtype = f32
every output finite
each row squared norm within 1e-5 of 1

The serialized output binds:

integration release;
checkpoint digest;
requested/selected device and fallback;
input, hidden, and output shapes;
dtype;
unit-normalized values.

This is an output contract, not just a tensor. Downstream retrieval can reject an embedding whose model/preprocessing identity or dimension differs from the index.

A provider-neutral adapter

Do not let the application depend directly on TinyCandleEncoder. Adapt it to the capability trait defined earlier:

struct CandleEmbeddings {
    model: Arc<TinyCandleEncoder>,
    model_identity: ModelIdentity,
}

impl Embeddings for CandleEmbeddings {
    async fn embed(&self, request: EmbedRequest)
        -> Result<EmbedResponse, CapabilityError>
    {
        // 1. validate request and preprocessing identity
        // 2. acquire an admission permit
        // 3. execute the bounded local model outside the async reactor
        // 4. map typed local errors without erasing them
        // 5. return vectors plus model/device/checkpoint receipts
        todo!()
    }
}

The actual preprocessing must transform text/image/audio/video input into the four-feature fixture only in a test adapter. A real encoder owns a versioned tokenizer or media preprocessor. Never present arbitrary handcrafted features as if they were a semantic foundation-model embedding.

Async services and blocking inference

Rust async does not make tensor kernels non-blocking. CPU inference can occupy a core; device calls can block during allocation, transfer, synchronization, or error handling. Running unbounded CPU inference directly on a Tokio reactor delays unrelated timers, sockets, and cancellation.

A service boundary needs:

request validation
  → weighted admission permit
  → bounded batching queue or dedicated worker
  → inference
  → output validation
  → permit release and receipt

Options include:

a dedicated inference thread per model/device;
a fixed worker pool;
spawn_blocking under a strict semaphore for modest CPU work;
a local inference service communicating through bounded channels;
a separate process when crash/resource isolation matters.

Do not put a semaphore inside spawn_blocking; many tasks may already consume blocking-pool slots while waiting. Acquire admission first. Cancellation after kernel launch may not abort device work, so distinguish “caller stopped waiting” from “compute was canceled.”

CPU thread pools can multiply unexpectedly: request workers × Rayon threads × BLAS threads × model replicas. Pin and measure the complete topology. More replicas are not free when they duplicate weights or contend for memory bandwidth.

Batching and memory

Batching amortizes dispatch and improves matrix utilization, but increases queueing and memory. Specify:

maximum items and total tokens/pixels/samples;
maximum queue delay;
compatible model, device, dtype, and preprocessing identity;
padding/bucketing policy;
fairness between tenants and request sizes;
cancellation handling;
OOM recovery behavior.

For a dense layer, activation values scale roughly with:

batch × positions × hidden_dimension

Transformer attention and K/V caches have different scaling. Image/video inputs can dominate before the model. Use weighted permits based on estimated peak bytes or compute, not only request count.

Dynamic batching changes latency distribution. Measure:

queue wait;
host preprocessing;
host-to-device transfer;
kernel execution;
device-to-host transfer;
output validation;
end-to-end latency;
actual batch fill and padding waste.

An OOM is not a normal cue to retry the same batch indefinitely. Lower a known-safe batch, evict under an explicit policy, route elsewhere, or fail with a typed resource error.

Training is a different product path

Inference needs immutable parameters, bounded inputs, forward execution, and checked outputs. Training additionally needs:

trainable Var ownership;
gradient graph and backward pass;
optimizer state, often multiple parameter-sized buffers;
gradient accumulation and zeroing semantics;
mixed-precision loss scaling when applicable;
data shuffling and reproducibility state;
checkpoint/resume consistency;
validation, early stopping, and artifact promotion;
protection against poisoned or unauthorized training data.

A service that occasionally fine-tunes adapters should not mutate the parameters currently serving requests. Train a versioned candidate artifact, evaluate it, then atomically promote or route it. Keep optimizer checkpoints separate from inference bundles.

Error handling and observability

Candle returns a structured error type, and its error-management guide describes adding context and optional backtraces. At the application boundary, preserve causal information but map it to stable categories:

invalid policy/input
artifact too large
artifact malformed
parameter schema mismatch
unsupported backend/dtype/operator
resource exhausted
kernel/backend failure
invalid output
internal invariant violation

Do not send raw filesystem paths, model URLs with credentials, tensor payloads, or driver dumps to clients. Put detailed causes in access-controlled logs with request/model/checkpoint/device identity and a bounded error chain.

Useful low-cardinality metrics include:

model-load success/failure by release and backend;
resident model count and bytes;
inference count/latency by model/backend/outcome;
queue wait and batch-size histograms;
fallback count;
invalid-output count;
OOM/resource-admission rejections;
backend reset or worker restart count.

Never label a metric by raw prompt, artifact digest for every user upload, or unbounded error text. Receipts belong in traces/run records or structured logs; aggregated metrics use bounded release labels.

Security and supply-chain boundary

A local model removes a remote inference dependency, not the attack surface. Review:

the Candle, safetensors, tokenizer, image/audio, and native transitive dependency graph;
exact source registries/git revisions and lockfile;
advisories, licenses, duplicate versions, and native build scripts;
model publisher identity, digest, signature, and license;
cache directory permissions and symlink/path traversal;
decompression and tensor-allocation bounds;
GPU sharing and tenant isolation;
model-induced output risk;
logs and crash dumps that may contain inputs or weights.

The companion intentionally uses safe buffered loading. If a later optimization introduces unsafe memory mapping, isolate that small boundary, document invariants, test file mutation/truncation behavior where practical, and retain the preflight parser/identity checks.

Treat weights as data with authority. A model can encode behavior, memorized material, or a backdoor even when the container format cannot execute code.

Performance method

Benchmark one question at a time:

pin exact source revision, lockfile, model/checkpoint digest, backend, device, driver, dtype, thread settings, and power mode;
separate cold artifact read, parse, device upload, graph warm-up, and steady-state inference;
use realistic shapes and batch distribution;
synchronize completion at accelerator timing boundaries;
collect enough samples for p50/p95/p99 and throughput, not one stopwatch value;
measure process RSS, device allocated/reserved memory, transfer bytes, and allocation failures;
validate outputs during benchmarks so a fast wrong kernel cannot “win”;
compare against a correctness baseline within declared absolute/relative tolerances;
test concurrency and queueing separately from single-request kernel speed;
publish the environment and raw samples.

Warm-up is not permission to discard arbitrary slow samples. Report cold-start behavior if real workers autoscale or models are loaded on demand.

Failure investigations

CPU is correct, accelerator is wrong

Freeze the same checkpoint bytes and input tensors. Then:

capture CPU and accelerator backend/library/device receipts;
compare parameter digests and post-load dtype;
compare outputs after each layer, beginning with the first divergence;
test a minimal operator with the exact shape/layout/dtype;
materialize contiguous inputs to isolate stride handling;
promote critical reductions to f32;
synchronize before reading or timing outputs;
check unsupported-operation fallback and backend-specific fast-math settings;
use absolute and relative error tolerances appropriate to the dtype;
reduce to a reportable fixture before upgrading drivers or libraries.

Do not immediately change the model, quantization, batch size, and backend together.

Shapes pass, semantics are wrong

Check:

tokenizer/preprocessor release and special-token/template policy;
axis order and weight transpose convention;
normalization epsilon and axis;
image channel order, color space, scale, mean, and standard deviation;
audio sample rate, channel mixing, and windowing;
attention mask polarity and positional indexing;
pooling token/axis;
adapter activation and parameter prefixes;
output postprocessing and similarity metric.

Golden intermediate activations from an independently trusted implementation make these failures locatable.

Startup OOM

Separate:

checkpoint file bytes;
host parsed tensors;
converted host tensors;
device parameters;
allocator reservation;
temporary conversion/workspace buffers;
duplicated model replicas.

A buffered load plus dtype conversion plus upload can temporarily hold several copies. Streaming or mapping may reduce host duplication, but only after its safety and lifecycle contract is correct.

Latency suddenly doubled

First verify the device receipt. A silent CPU fallback, changed dtype, missing optimized kernel, new non-contiguous input, thread oversubscription, reduced batch fill, thermal throttling, or implicit per-request transfer is more likely than “Rust became slow.”

Verification in the companion

The chapter adds ten module tests and one complete-example test:

Test area	Proven property
real safetensors fixture	exact four-parameter, 23-value contract loads
golden forward	linear → ReLU → projection → unit normalization yields exact expected values
input validation	empty, oversized, wrong-width, and non-finite batches fail before inference
byte budget	oversized checkpoint fails before parsing
parameter set	missing and extra tensors are rejected
shape contract	misshaped parameter is rejected
value contract	non-finite parameter is rejected
parser boundary	malformed safetensors returns a typed error
device policy	accelerator fallback occurs only when explicitly allowed and is recorded
layout	transpose is non-contiguous and explicit materialization is contiguous
identity	changing checkpoint content changes its digest

Run:

cargo test -p mosaic-ml candle_runtime::tests
cargo test -p mosaic-ml --example candle_model_pipeline
cargo clippy -p mosaic-ml --all-targets --all-features -- -D warnings

The fixture is generated through the real safetensors serializer and loaded through Candle’s real buffer loader. It is not JSON pretending to be a checkpoint.

Production acceptance gate

A new Candle-backed model/backend combination is not ready until all are true:

immutable model bundle and checkpoint digests are verified;
license and artifact-source policy pass;
tokenizer or media preprocessor identity is bound to the model;
architecture release and exact parameter schema match;
shapes, dtypes, counts, bytes, and memory plan remain within checked bounds;
compiled and selected device/backend are recorded with no accidental fallback;
unsupported operators/dtypes fail during readiness, not the first user request;
CPU/reference golden vectors and layer-local comparisons pass within declared tolerances;
invalid, empty, maximum-size, non-finite, and adversarial inputs are tested;
output shape, dtype, finiteness, normalization, and semantic postconditions are checked;
cold load, warm throughput, p95/p99 latency, peak memory, and batch behavior meet SLOs;
admission, cancellation, timeout, drain, and worker recovery behavior are exercised;
errors/logs/traces are useful without exposing inputs, secrets, or weights;
dependency advisories, licenses, sources, duplicates, and native build scripts are reviewed;
rollback keeps the prior artifact and routing configuration available.

Exercises

Exercise 1: prove the second row by hand

Using input [-1,1,1,0], calculate hidden values, projection values, norm, and normalized output. Compare them to [0.83205026, 0.5547002].

Exercise 2: make zero norm explicit

Create parameters/input that produce [0,0] before normalization. Add either a typed ZeroNormOutput failure or a documented epsilon. Test the exact boundary and explain the effect on cosine similarity.

Exercise 3: version optional parameters

Define schema v2 with an optional, declared projection scale. Reject it under v1, accept exactly one correctly shaped projection.scale under v2, and still reject unrelated extras.

Exercise 4: add an embedding adapter

Implement the provider-neutral Embeddings capability for the fixture. Acquire a weighted permit before entering spawn_blocking, propagate request/model/device receipts, and prove a canceled caller cannot admit unbounded background work.

Exercise 5: compare dtypes

On hardware that supports the path, load or convert the fixture to a reduced dtype. Record storage, compute, and output dtype. Compare every layer against the f32 baseline with explicit absolute and relative tolerances. Do not use exact floating-point equality.

Exercise 6: reproduce a layout regression

Benchmark a sufficiently large matrix operation with an appropriate contiguous input and a semantically identical non-contiguous view. Include materialization cost rather than timing only the final kernel.

Exercise 7: build a backend CI matrix

Design CPU, CUDA, and Metal jobs with exact runner/toolchain requirements. State which jobs are required for a portable change and which are required before publishing each platform artifact.

Exercise 8: harden a mapped loader

Write the safety argument for a memory-mapped model store: directory ownership, descriptor acquisition, identity verification, immutability, mapping lifetime, truncation behavior, and retirement. If any invariant cannot be guaranteed, keep buffered loading.

Exercise solutions

Solution 1

The first layer produces [-1,1,1]; ReLU makes [0,1,1]. Projection plus bias is:

z₀ = 0 + 0 + 0.5 + 0.1 = 0.6
z₁ = 0 + 1 - 0.5 - 0.1 = 0.4

The norm is sqrt(0.36 + 0.16) = sqrt(0.52) ≈ 0.7211103. Division gives approximately:

[0.8320503, 0.5547002]

Solution 2

A typed failure preserves the mathematical contract “every returned row is a unit vector.” An epsilon policy instead returns a near-zero vector whose norm is not one and must be documented as a different output contract. For retrieval embeddings, rejecting usually exposes a model or preprocessing fault more clearly.

Solution 3

Represent accepted parameter sets as schema versions. Validate the complete actual set against the selected version before VarBuilder construction, then validate projection.scale as shape [2], accepted dtype, finite values, and counted resource use. Do not infer v2 merely because the extra tensor exists; the bundle manifest chooses the schema.

Solution 4

The safe order is validate → acquire permit → enter bounded blocking execution → validate output → release. If the caller cancels after launch, retain accounting until the worker actually finishes. Return the same provider-neutral response fields as remote or alternate local adapters so callers do not branch on Candle.

Solution 5

Use the exact checkpoint/input and retain the f32 output as reference. Compare:

absolute_error = |candidate - reference|
relative_error = absolute_error / max(|reference|, floor)

Choose tolerances from task quality and measured backend behavior, not merely to make a test pass. Also compare unit norms and downstream ranking stability.

Solution 6

Report three measurements: view construction, materialization, and the operation. A view may be nearly free while .contiguous() dominates. If the kernel accepts strided input, compare complete end-to-end paths and validate identical outputs.

Solution 7

The CPU job remains the portable required gate. The CUDA artifact is publishable only after its pinned CUDA job builds, runs golden vectors, exercises OOM/fallback policy, and records driver and device. The Metal artifact requires the corresponding macOS job. A documentation-only or CPU change need not force unavailable toolchains into one --all-features job, but shared tensor code must still run every supported backend gate before release.

Solution 8

Open only within a controlled model-store directory, reject symlinks/path escape, verify a stable descriptor-backed file, require immutable ownership/permissions, hash the exact bytes, map read-only, and retain the file/mapping for every tensor lifetime. Deployment must never replace a mapped file in place; publish a new digest-addressed file and retire the old mapping after users drain. Otherwise use the simpler bounded buffer.

Review checklist

Before continuing, you should be able to answer:

What information does shape omit that layout and dtype supply?
Why can transpose be cheap while .contiguous() is expensive?
Which axes does Linear assign to [out,in]?
Why are raw success and correct output different properties?
What does safetensors prevent, and what does it not validate?
Why are exact parameter names checked before VarBuilder construction?
Why must accelerator fallback appear in a response/run receipt?
Which dtype describes storage, compute, accumulation, and output in your system?
Why can async Rust still block on local inference?
What must an accelerator benchmark synchronize?
When are gradients and optimizer state necessary?
Which identities must accompany an embedding before it enters an index?

If any answer is vague, rerun the fixture, alter one invariant, and predict the typed failure before reading the test.

References

What comes next

You now own the lowest visible model boundary: bytes become validated parameters; parameters and inputs become tensors on a named device; tensor operations become a checked result with exact identity. The next chapter uses mistral.rs to study higher-level local serving for text and multimodal generation—model loading, chat templates, paged context, quantization, device mapping, streaming, and server isolation—without abandoning these underlying contracts.