mistral.rs and Multimodal Local Serving

Running a local model is not the same as operating a local inference service.

mistralrs serve -m <model> can make a model answer on a laptop. A production caller still needs to know which binary and model bytes answered, whether the requested accelerator was active, whether a multimodal file came from an authorized artifact or an attacker-controlled URL, whether the route is actually ready, whether a stream ended cleanly, and whether disconnecting a client stopped compute. “OpenAI compatible” describes a useful wire shape; it does not create those application invariants.

This chapter places mistral.rs in a separate local service process. The companion does not download multi-gigabyte weights during CI and does not label scripted output as inference. It implements the part that must remain deterministic:

a pinned binary/model launch contract expressed as argument boundaries, never shell text;
loopback-only networking with the UI and agentic tools absent;
exact model-route readiness, not merely process liveness;
digest-addressed image, audio, and video staging;
a bounded OpenAI chat-completion request with ordered content parts;
a strict SSE consumer that enforces response/model identity, one choice, bounded output, terminal usage, a supported finish reason, and one [DONE];
receipts binding launch, request, evidence locators, and stream bytes.

The worked release is pinned to mistral.rs 0.8.2. mistral.rs evolves quickly, so the exact version and its generated OpenAPI document are part of the contract, not an incidental detail. Also note that mistral.rs is an independent project and is not affiliated with Mistral AI.

Learning objectives

By the end you will be able to:

choose between embedding the Rust SDK, mounting the server router, and running a separate inference process;
pin the mistral.rs binary, compile/runtime backend, model revision, and complete local bundle;
explain why a Hugging Face model ID alone is not immutable;
build a launch argument vector without shell interpolation;
separate liveness, exact-route readiness, and semantic model acceptance;
understand text, image, audio, and video support as model-family-specific capabilities;
construct ordered OpenAI content parts without losing evidence artifact and locator identity;
prevent server-side file:// access and remote URL fetching from becoming SSRF or file-read primitives;
bound media count, encoded bytes, prompt bytes, JSON bytes, context, output tokens, and concurrent sequences independently;
consume chat-completion SSE as a protocol state machine rather than concatenating every data: line;
handle keep-alives, chunk identity, finish reasons, usage, named events, and [DONE];
distinguish connection cancellation from compute cancellation and resource reconciliation;
reason about continuous batching, paged attention, prefix caching, K/V pool sizing, and head-of-line effects;
choose plain safetensors, GGUF/GGML, UQFF, or in-situ quantization deliberately;
configure CPU, CUDA, Metal, automatic device mapping, and per-layer topology without silent operational drift;
operate authentication, TLS, logging, metrics, model reload, and rollback outside the happy path;
design an end-to-end live-model acceptance test without putting a model download in every unit test.

Prerequisites and dependency order

Complete Candle fundamentals first. It explains parameter schemas, tensor shapes/layout, dtype, device selection, and why a successful graph is not proof of a correct model. Review inference topologies to compare process boundaries, quantization and device placement for memory planning, and provider-neutral capability traits so callers do not become coupled to mistral.rs.

The previous multimodal chapters define the evidence artifacts and locators staged here. A media URL is transport; its digest and source locator are the evidence identity.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example mistral_multimodal_service

The example emits one JSON report. Its launch identity is:

b9d9ab16a075d8d225cd891a7747fcf219af636d9940822fcafacf9fdb82a5d0

The structured launch arguments include:

serve
--model-id /srv/mosaic-models/2222…2222
--format plain
--host 127.0.0.1
--port 1234
--no-ui
--access-log-format json
--max-seqs 4
--max-seq-len 8192
--max-batch-size 4
--paged-attn off
--prefix-cache-n 4
--max-num-images 4
--max-edge 1024
--cpu

The request contains three ordered, staged artifacts:

image/png → file:///srv/mosaic-evidence/<image-digest>.png
audio/wav → file:///srv/mosaic-evidence/<audio-digest>.wav
video/mp4 → file:///srv/mosaic-evidence/<video-digest>.mp4
text      → Explain the same cancellation event across all evidence.

It is 735 bytes of JSON, requests a maximum of 64 output tokens, sets temperature 0, seed 7, streaming, and streamed usage. The example then validates a realistic SSE fixture and assembles:

Evidence spans text, image, audio, and video.

Only after seeing one stop terminal chunk, usage 128 + 9 = 137, and [DONE] does the consumer return success.

This run proves the integration protocol. It does not prove that Gemma 4 generated that sentence. A live acceptance gate later substitutes an actual pinned server and model while keeping the same request/response validators.

Three integration topologies

High-level Rust SDK

The mistral.rs Rust SDK can embed the engine:

let model = mistralrs::ModelBuilder::new("Qwen/Qwen3-4B")
    .with_max_num_seqs(4)
    .build()
    .await?;

This gives type-level access and avoids an HTTP hop. It also brings the engine’s large dependency graph, native/backend features, model memory, worker lifecycle, and potential failure modes into the application process. Choose it when one binary, one scaling unit, and one trust boundary are actually desirable.

Mounted server core

mistralrs-server-core can mount routes in an existing Axum application and hook the server lifecycle. This is useful when the default body/CORS/middleware behavior must be changed at the same router. It also tightly couples application upgrades and inference-server upgrades and shares the process’s failure/resource boundary.

Separate local service

The companion chooses:

product API / harness
        │ provider-neutral capability
        ▼
authenticated local client
        │ loopback/private HTTP
        ▼
version-pinned mistral.rs process
        │
        ├── immutable model store
        └── read-only staged evidence store

This adds serialization and a local network hop. It permits independent:

model/server rollout and rollback;
accelerator scheduling;
memory limits and crash recovery;
concurrency/batching;
application and inference dependency graphs;
health and readiness;
multiple application replicas sharing or not sharing one model worker.

It is not automatically safer. An unauthenticated service bound to all interfaces is a new remote-control surface.

Pin the serving identity

Binary identity

The binary contract records:

exact mistral.rs version;
SHA-256 of the installed binary or immutable container image digest;
build target and enabled CPU/CUDA/Metal features;
Rust/toolchain and native toolkit where built from source;
driver/runtime compatibility observed on the target;
startup configuration digest.

The companion accepts exactly 0.8.2 and a lowercase 64-hex digest. "latest" fails. In a real release, upgrade the constant only after the API, golden outputs, load behavior, dependency audit, and deployment gate pass.

Prebuilt installers are convenient for development. Production should verify the downloaded artifact, source provenance, or pinned container digest rather than piping changing network content into a shell.

Model identity

The official production guidance warns that model IDs resolve to a changing main revision unless pinned. The companion therefore records:

pub struct VerifiedModelSource {
    pub public_model_id: String,
    pub revision_commit: String,
    pub bundle_sha256: String,
    pub local_model_dir: String,
}

public_model_id is human context. revision_commit identifies the upstream snapshot. bundle_sha256 identifies the reviewed local bundle manifest. The server receives only a digest-addressed directory:

/srv/mosaic-models/<bundle-sha256>

That directory contains the exact weights, configuration, tokenizer, chat template, processor, licenses, and manifest verified in the model-artifact chapter. A mutable cache entry or repo ID is not sufficient for a regulated release.

Model route identity

The request’s model field controls routing when multiple models are loaded. "default" is convenient but binds behavior to server configuration. The companion sends the exact expected route and requires every stream chunk to repeat it.

The route is still not the complete artifact identity. /v1/models reports serving status; the deployment receipt binds the route to the binary and bundle. Both are required.

Launch with arguments, not a shell

The module returns Vec<String>:

let args = serve.launch_args();
Command::new(verified_binary_path).args(&args).spawn()?;

Do not create:

Command::new("sh").arg("-c").arg(format!("mistralrs serve -m {model}"));

Even a locally administered model string can contain whitespace or shell metacharacters after a configuration mistake. Argument boundaries also make the exact launch receipt serializable and testable.

The worked plan is deliberately conservative:

loopback host;
built-in UI disabled;
no MCP port;
no agent, search, code execution, shell, skills, or dynamic LoRA mutation;
JSON access logs;
four sequences and batch size four;
explicit context, cache, image, and CPU policies;
paged attention off because the CPU fixture forbids pretending it is active.

This is a teaching plan, not a performance recommendation for a large Gemma model on CPU. A production deployment selects a quantized artifact and hardware based on measured quality, latency, throughput, and memory.

Network and authentication

As of the pinned release, the HTTP API semantics state that the server has no built-in authentication. It accepts compatibility authorization headers but does not validate them. Therefore:

bind to 127.0.0.1 for a same-host client;
or place the service on a private network behind an authenticating, authorizing TLS proxy;
enforce tenant/model/action policy before forwarding;
limit source addresses and egress;
never interpret “Bearer header accepted” as authenticated.

The documented default server body limit is 50 MB and default CORS allows any origin; those controls are not CLI-configurable in the pinned surface. If those defaults violate the threat model, use a reverse proxy with a smaller limit and strict origins or embed the server-core router with custom middleware.

Loopback is a transport boundary, not an authorization boundary on a multi-user host. Protect the socket, model files, staged media, logs, and process account.

Liveness is not readiness

The root/health route returning 200 means the HTTP server is listening. The official production checklist explicitly says it does not verify model load.

The companion reads /v1/models, bounds the response, requires object "list", caps the inventory at 64, finds the exact route exactly once, and requires:

{
  "id": "/srv/mosaic-models/<bundle-digest>",
  "status": "loaded"
}

unloaded and reloading are not ready. A duplicate exact ID is a protocol failure. A different loaded model does not make this route ready.

Use three probes:

Probe	Question	Failure action
startup	Did the process finish initial load before its generous deadline?	restart/diagnose artifact or capacity
liveness	Is the server process responding?	restart only after a conservative failure threshold
readiness	Is the exact required model route `loaded` and accepted by release policy?	stop new routing; do not necessarily restart

Semantic readiness is a fourth gate: send a tiny known request and verify output contract when a release starts. Do not use an expensive generation request as a high-frequency Kubernetes probe.

Multimodal support is a capability matrix

mistral.rs supports text, vision, audio, video, embeddings, speech, and image generation across different model families. That does not mean every model accepts every modality. The current supported-model reference is the source of truth for a pinned release.

Examples in the current docs include Qwen3-VL for image/video and Gemma 4 for image/audio/video. Audio support is model-specific. Image-generation, speech-generation, embedding, and chat-understanding endpoints are different capabilities even if one engine serves them.

Create a release matrix:

Release route	Text	Image input	Audio input	Video input	Text output	Other output
exact multimodal chat release	yes	evaluated	evaluated	evaluated	yes	no
exact embedding release	yes	maybe	maybe	maybe	vectors	no
exact speech release	text/audio	no	model-specific	no	maybe	audio
exact diffusion release	text	maybe	no	no	metadata	image

The word “supported” means the engine can load a model path. “Accepted” means your exact model/backend/preprocessor passes quality, safety, performance, and failure evaluations.

Stage evidence; do not accept arbitrary URLs

The HTTP content-part format uses:

{"type":"image_url","image_url":{"url":"..."}}
{"type":"audio_url","audio_url":{"url":"..."}}
{"type":"video_url","video_url":{"url":"..."}}

The server supports file://, http(s)://, and data: URLs. Exposing those forms directly to untrusted callers creates risk:

file:// can attempt reads under the inference service’s filesystem permissions;
HTTP can perform server-side requests to internal services, metadata endpoints, or huge/slow resources;
redirects and DNS rebinding complicate allowlists;
data URLs inflate JSON and decoded allocation;
repeated URLs can bypass product-level artifact accounting;
remote content can change between evaluation and replay.

The companion accepts none of them from the caller. An earlier evidence pipeline:

acquires bytes under URL/network policy;
checks encoded and decoded bounds;
hashes and stores immutable content;
records modality, media type, locator, and provenance;
publishes an operator-owned read-only staged file;
constructs the URL from the digest and known format.

For example:

file:///srv/mosaic-evidence/<sha256>.png

The request receipt retains ordered artifact and locator digests. The server sees the file needed for preprocessing; the harness retains the source-restorable evidence identity.

Mount only the staging directory read-only into the inference process. Never mount a user’s home, workspace, object-store credentials, Docker socket, or entire artifact store for convenience. Remove staged references only after in-flight users drain.

Preserve content-part order

Multimodal message content is an ordered list. This:

before image → after image → “What changed?”

is not equivalent to:

after image → before image → “What changed?”

The companion preserves:

image → audio → video → text

and stores artifact/locator digests in the same order. Do not sort content parts by digest for canonicalization. Canonicalize the serialized record without erasing semantic order.

The request validator separately caps:

prompt bytes;
media-part count;
sum of encoded media bytes;
output tokens;
final JSON bytes;
admitted modalities.

These are independent. Three small files can fit the count but exceed bytes; one data-heavy prompt can fit context tokens under one tokenizer but exceed transport policy; a tiny JSON request can point to a huge staged video. The earlier decode policies and service-side media limits still apply.

The exact request contract

The generated request is equivalent to:

{
  "model": "/srv/mosaic-models/<bundle-digest>",
  "messages": [{
    "role": "user",
    "content": [
      {"type":"image_url","image_url":{"url":"file:///srv/mosaic-evidence/<image>.png"}},
      {"type":"audio_url","audio_url":{"url":"file:///srv/mosaic-evidence/<audio>.wav"}},
      {"type":"video_url","video_url":{"url":"file:///srv/mosaic-evidence/<video>.mp4"}},
      {"type":"text","text":"Explain the same cancellation event across all evidence."}
    ]
  }],
  "max_tokens": 64,
  "temperature": 0.0,
  "seed": 7,
  "stream": true,
  "stream_options": {"include_usage": true}
}

Every control is explicit. Temperature zero does not guarantee identical output across different engine builds, kernels, model bytes, devices, batching schedules, or tokenizer templates. The seed is useful replay evidence within a fixed runtime boundary, not universal determinism.

The request receipt binds:

integration release
request ID
endpoint
serve identity digest
serialized request digest and byte length
exact model route
ordered artifact digests
ordered locator digests
maximum output tokens

Send the request ID as x-request-id. Current mistral.rs echoes or generates that header and uses it in access logs. Verify the returned value at the client boundary; the chapter fixture focuses on body protocol and records the intended ID.

Media preprocessing is part of the model

The engine can decode and preprocess media, but its choices become behavior:

image resize, maximum edge, aspect handling, color conversion, normalization, and patching;
audio format decode, sample rate, channel conversion, windowing, and feature extraction;
video container decode, frame sampling, resize, and ordering;
insertion of modality tokens and chat-template positions.

The CLI exposes --max-num-images, --max-edge, and --max-image-length controls. Current mistral.rs documentation says non-GIF video needs FFmpeg and that per-request video frame-sampling controls are not currently exposed. That means a product needing exact evidence-frame selection should preprocess/synthesize the accepted frame set earlier or evaluate the engine’s fixed behavior—it should not assume source-video citation precision from a whole-file request.

WAV, MP3, FLAC, and OGG are documented as native audio inputs in the pinned docs. Still normalize to an accepted, evaluated artifact when exact replay matters.

Bind the processor/config/chat-template files in the model bundle. A model revision with a different processor can change quality and resource cost without changing the application JSON.

Consume SSE as a state machine

With stream: true, chat completions use Server-Sent Events. Default data: events carry OpenAI-shaped chat chunks, keep-alive comments may arrive, and:

data: [DONE]

terminates the stream. mistral.rs may also interleave named agentic events when those features are enabled.

The companion’s non-agentic contract accepts:

zero or more comment-only keep-alive blocks
  → one or more default/message JSON chunks
  → exactly one terminal `stop` chunk with usage
  → exactly one [DONE]
  → end of bytes

For every JSON chunk it requires:

object chat.completion.chunk;
bounded, valid response ID;
same response ID, creation timestamp, and model route across chunks;
exactly one choice at index zero;
assistant role at most once;
bounded, NUL-free text;
no usage before the terminal chunk;
supported finish reason stop;
prompt_tokens + completion_tokens == total_tokens;
completion tokens within the requested maximum.

It rejects named agentic events because agentic tools are not in this release contract. A future agent adapter must parse each named event into a separate typed state machine; it must not discard approval or file events as if they were text.

Why `[DONE]` matters

A TCP EOF after partial text can look like a plausible answer. Without a terminal chunk and [DONE], the caller cannot distinguish success from:

server crash;
proxy timeout;
client/network interruption;
parser truncation;
process restart;
upstream stream bug.

Persist partial output as a failed/incomplete attempt if useful, but never promote it to a complete answer.

Finish reasons

stop is the fixture’s accepted success. A length/token-cap reason is not necessarily an engine failure, but it is an incomplete task result unless the product contract explicitly accepts it. Tool-call and content-filter reasons require different output types. Map each known reason; reject unknown values under the pinned adapter until reviewed.

Forward compatibility

The wire parser accepts irrelevant extra JSON fields through its selected typed view but strictly validates fields it uses. This permits benign OpenAI-compatible extensions while preventing model, choice, role, terminal, or usage drift. The deployment gate fetches the server’s generated OpenAPI document for each upgrade and updates fixtures intentionally.

Backpressure and cancellation

Streaming can move the bottleneck from generation to the client. A slow browser, proxy, or downstream consumer must not create an unbounded queue of chunks.

Use:

HTTP body stream
  → bounded SSE decoder buffer
  → bounded typed-event channel
  → consumer/persistence

Cap:

response headers;
individual line/event bytes;
total wire bytes;
event count;
assembled text/tool bytes;
idle interval and total deadline;
buffered downstream events.

The fixture consumes an already bounded byte slice to make protocol invariants deterministic. A live client applies the same state machine incrementally across arbitrary TCP chunk boundaries; never assume one read equals one SSE event.

Dropping the HTTP connection may notify the server, but do not assume it cancels a kernel already enqueued. Treat these as separate states:

caller canceled waiting
connection closed
server accepted cancellation
sequence removed from scheduler
device work actually stopped
usage reconciled

Retain admission accounting until the local worker or server confirms termination, or until a bounded orphan-reconciliation path takes ownership.

Continuous batching and sequence admission

mistral.rs supports continuous batching. --max-seqs caps sequences running concurrently; extra requests wait. It is not a complete product admission policy:

a long multimodal prompt costs more than a short text prompt;
image/video preprocessing may happen before sequence scheduling;
tenants need fairness;
queued requests still consume sockets/memory/deadlines;
one very long generation can affect tail latency.

The product API should acquire a weighted permit before calling the inference server. The server’s sequence cap is a second containment layer.

Distinguish:

--max-seqs: concurrently scheduled sequences;
--max-batch-size: sizing input for automatic device mapping;
product-side maximum active/queued work;
dynamic microbatch size at each generation step.

Measure actual queue wait, prefill batch composition, decode batch composition, prompt/output tokens, media preprocessing, and per-tenant latency.

Paged attention and K/V memory

Standard allocation can reserve a contiguous K/V cache per sequence at maximum context. Paged attention splits K/V storage into fixed blocks allocated from a shared pool, reducing wasted reservation under varied sequence lengths and supporting higher continuous-batching utilization.

The current CLI supports:

--paged-attn auto|on|off;
one of context length, memory MiB, or memory fraction for pool sizing;
block size;
K/V cache type.

The pinned docs state auto enables paged attention on CUDA and disables it on Metal/CPU. Do not log “paged attention enabled” merely because the argument was auto; capture the resolved backend and runtime configuration.

Paged attention does not eliminate K/V cost:

active blocks × block tokens × layers × K/V heads × head dimension × 2 × bytes/value

It changes allocation/fragmentation and sharing. An undersized pool rejects or queues work; an oversized fraction can starve weights, workspaces, vision encoders, or other processes.

Prefix caching

Prefix caching can reuse computation for identical tokenized prefixes, including multimodal prefixes where supported. Correctness requires identity across:

model weights and adapters;
tokenizer and chat template;
full ordered messages/content parts;
media preprocessing and artifact bytes;
relevant generation/runtime state.

Never key a cache only by visible prompt text. Two images with the same instruction are different prefixes. Treat cross-tenant reuse as a potential timing/privacy channel. Disable or partition the cache if the threat model requires it.

--prefix-cache-n 0 disables the cache; the fixture keeps four entries so the choice appears in the serving identity.

Formats and quantization

The current server accepts plain safetensors, GGUF/GGML, and UQFF-related paths, with several prequantized and in-situ quantization mechanisms.

Keep these concepts separate:

plain: upstream parameter tensors, possibly reduced precision;
GGUF/GGML: packaged/quantized formats with metadata and tokenizer/model conventions;
UQFF: mistral.rs universal quantized artifacts;
ISQ: load a model then quantize selected weights in situ;
--quant: a convenience front door that may select a published prebuilt UQFF or apply ISQ;
topology: per-layer device and quantization decisions.

A numeric “4-bit” label does not identify the algorithm, group size, calibration, exempt layers, compute dtype, or kernel. Record the exact artifact and quantization metadata and evaluate:

perplexity/task quality;
instruction/tool/structured-output compliance;
multimodal grounding;
calibration and safety slices;
prefill/decode throughput;
peak load and steady-state memory;
cold-start conversion time.

For reproducible production startup, prefer a precomputed, verified quantized artifact over repeating costly ISQ at every boot. If runtime quantization is chosen, bind its release, parameters, input digest, output artifact digest, and hardware-dependent behavior.

Device mapping and topology

The CLI can force CPU, automatically map layers, accept device-layer assignments, or read a topology YAML. Topology can select devices and ISQ by numbered layer range or weight-name regex.

Example from the official model:

0-16:
  device: cuda[0]
  isq: q4k
16-32:
  device: cuda[1]
  isq: q4k
32-40:
  device: cpu
  isq: q8_0

This is powerful and easy to mismeasure. CPU-offloaded layers transfer activations across the bus; uneven layers create device imbalance; one slow link can dominate. A topology file is executable deployment policy and belongs in the release manifest with a digest.

Run mistralrs doctor on the target host to inspect backend capability. Use mistralrs tune as a starting recommendation, not proof: the current production guide says tuning estimates from configuration and does not load/benchmark the model. Then benchmark the exact workload.

Chat templates, structured output, and tools

Chat models consume token sequences, not abstract roles. mistral.rs may auto-detect a chat template, or the operator can supply one. Pin the resolved template. A change can alter special tokens, role ordering, tool syntax, reasoning markers, context length, and safety behavior.

Structured output and tool calling deserve their own capability contract. Do not parse JSON from ordinary free text merely because it often works. Use the engine’s grammar/schema support where accepted, then validate the result again in the application.

Agent mode can enable web search, Python execution, shell execution, tool dispatch, skills, MCP, sessions, and approvals. Those features materially expand authority. The server CLI warns that code and shell execution allow arbitrary execution. They are absent from this chapter’s launch plan.

If enabled later:

create an explicit tool allowlist per tenant/task;
use authenticated approval;
sandbox with filesystem/network/process/resource policy;
separate model suggestion from authorization;
bound rounds, output, and wall/CPU time;
persist complete tool-call/result traces;
treat produced files as untrusted artifacts;
never rely on the model to self-police.

Observability and sensitive logs

The pinned service exposes:

structured tracing/startup/access logs;
x-request-id propagation;
Prometheus request counters, duration histograms, in-flight gauges, and body-size histograms;
model inventory/status.

The metrics use matched route patterns rather than raw URLs, which controls cardinality. Product telemetry should correlate:

product request/run ID
→ local HTTP request ID
→ exact route
→ serve identity
→ request digest
→ response/stream digest

Do not label metrics with raw prompts, media paths, artifact digests, or response IDs.

The official observability guide warns that -l/--log <path> records full model requests and responses, not just access metadata. Keep it off unless storage, retention, access, deletion, and redaction policy explicitly allow payload logging.

Add application metrics for:

readiness transitions;
admission rejection and queue wait;
first-event latency and inter-event gaps;
complete/incomplete/protocol-failed streams;
finish reasons and usage reconciliation;
modality/preprocessing latency;
fallback or route mismatch;
model load/reload/rollback.

Lifecycle, rollout, and rollback

Model startup can take minutes. Give startup a separate generous deadline while keeping liveness timeouts short once ready.

A safe rollout:

stage and verify a new digest-addressed binary/model/config;
start a new worker without traffic;
wait for exact-route loaded;
run golden protocol, semantic, multimodal, resource, and safety smoke tests;
mark the worker ready;
shift a small traffic cohort;
compare quality, failures, TTFT, throughput, memory, and cost;
increase traffic only within release gates;
stop new admissions to the old worker;
drain streams/sequences to a deadline, then terminate and record incomplete work;
retain the previous immutable release for rollback.

The process health route alone cannot gate promotion. Neither can one visually pleasing answer.

Dynamic model/LoRA mutations complicate rollback and do not necessarily survive restart. The pinned production docs describe runtime LoRA safeguards and warn that mutation endpoints have no built-in authentication. The conservative path is immutable preload + new process release.

Failure investigations

Health is green, requests fail

Check /v1/models for the exact route. Then inspect:

load/reload status and startup logs;
model path permissions and bundle completeness;
binary backend versus target hardware;
model architecture support in this exact release;
dtype/quantization/kernel support;
K/V/weight/workspace memory;
chat template/tokenizer/processor files.

Do not restart in a tight loop while repeatedly downloading or quantizing the same bad artifact.

First token is slow

Split:

product queue wait;
HTTP connection/serialization;
media read/decode/preprocessing;
tokenization/template rendering;
scheduler queue;
prefill;
first decode step.

For multimodal input, media preprocessing and vision/audio towers can dominate. Prefix caching helps only when the exact evaluated prefix matches.

Stream text stops midway

Look for terminal finish reason, usage, and [DONE]. If any is missing, classify incomplete. Correlate request ID across proxy and server logs. Check:

proxy idle/total timeout;
client backpressure;
server OOM/restart;
response/event byte limits;
malformed named event handling;
cancellation.

Do not repair missing suffixes by asking another model and call the result the same run.

CPU unexpectedly serves production

Compare the deployment’s compiled backend, doctor output, launch plan, and observed device receipt. An automatically selected CPU path may be functionally correct but violate every latency and capacity assumption. Fail readiness if the required backend is absent.

Image works, video fails

Check the exact model’s modality matrix, FFmpeg availability, codec/container support, server file permissions, encoded/decoded bounds, frame count/dimensions, and preprocessing logs. Reduce to a small immutable clip while preserving the failure. Do not assume image support implies video.

Output quality changed after “only” an upgrade

Diff:

mistral.rs binary/source/features;
model revision and every bundle digest;
tokenizer/processor/chat template;
quantization and topology;
dtype/backend/driver;
sampling defaults;
media preprocess configuration;
prefix cache/adapters;
prompt and content-part order.

Replay the frozen evaluation suite and compare intermediate token counts, not only final prose.

Verification in the companion

The chapter adds twelve module tests and one complete-example test:

Test area	Proven property
launch arguments	pinned, loopback, no UI/agent, explicit CPU/paged-attention, no shell-shaped argument
serve identity	changing cache/config changes identity; version/digest/revision errors fail
media staging	only typed PNG/WAV/MP4 digest-addressed paths are constructed
request order	image → audio → video → text and artifact/locator order survive serialization
request bounds	modality, count, media bytes, output tokens, and JSON size fail independently
readiness	exact route must appear once with `loaded`
stream success	keep-alive + chunks + usage + `[DONE]` assemble one checked receipt
terminal state	missing finish, usage, or `[DONE]` fails
chunk stability	model, choice index, and response ID drift fail
named events	undeclared agentic events fail closed
stream bounds	wire bytes, event count, and output bytes are independent
identities	serving, request, and stream content changes alter their digests

Run:

cargo test -p mosaic-ml mistral_serving::tests
cargo test -p mosaic-ml --example mistral_multimodal_service
cargo clippy -p mosaic-ml --all-targets --all-features -- -D warnings

The suite deliberately avoids a network/model dependency. A separate opt-in acceptance job owns:

verified mistral.rs binary
  → verified local model bundle
  → target CPU/CUDA/Metal host
  → readiness
  → frozen text/image/audio/video requests
  → exact protocol receipts
  → tolerance/task-based semantic evaluations
  → latency/memory/throughput gate

Live acceptance gate

Before routing real traffic, require:

exact binary/container digest and feature receipt;
doctor confirms the required backend;
model revision, bundle, tokenizer, template, processor, topology, and quantization digests;
immutable read-only model and evidence mounts;
loopback/private bind plus authenticated TLS proxy;
body/CORS/egress/file-access policy;
built-in UI and every unapproved agent/tool/mutation surface disabled;
explicit sequence, context, batch, K/V, prefix-cache, image-count, and image-size limits;
exact-route readiness, not health alone;
request-ID round trip;
text/image/audio/video golden and adversarial cases for the accepted matrix;
source-artifact/locator identity retained outside the wire URL;
complete-stream enforcement under disconnect, slow consumer, timeout, and process restart;
output shape/schema/finiteness/citation/safety validation for the selected capability;
cold load, TTFT, inter-token latency, throughput, queueing, CPU/RAM/VRAM, and OOM tests;
dependency/license/advisory/source/native-build review;
canary comparison and one-command routing rollback to the prior immutable release.

Exercises

Exercise 1: implement an incremental SSE decoder

Feed the fixture one byte at a time and at random chunk boundaries. Reuse the same event state machine, bound the line/event buffer, and prove results match whole-buffer parsing.

Exercise 2: model length termination

Add a typed IncompleteLengthLimited outcome for finish_reason = "length". Preserve partial text and usage without treating it as success. Write a retry policy that changes one budget only when the product permits it.

Exercise 3: authorize staging

Design a staging API that accepts tenant, evidence ID, artifact digest, locator digest, media type, and lease. Ensure a request cannot reference another tenant’s file and cleanup cannot remove an in-flight lease.

Exercise 4: add live header validation

Verify HTTP status, content type, x-request-id, and bounded headers before parsing SSE. Preserve a sanitized response-body prefix for non-2xx diagnostics without logging prompts or media URLs.

Exercise 5: compare embedded and service topology

For one target, calculate model-memory copies, process hops, failure domains, rollout units, required replicas, and accelerator sharing for SDK-embedded versus separate service deployment.

Exercise 6: evaluate quantization

Compare plain/BF16 with one exact UQFF or GGUF release. Freeze task cases and report quality slices, calibration, TTFT, decode speed, peak memory, and artifact load time. Decide from the gate rather than bit width.

Exercise 7: protect remote media

If product requirements force HTTP media URLs, design DNS/IP/redirect/port/scheme/size/time/content controls, an isolated fetcher, content hashing, and cache semantics. Explain why the inference process itself should still receive only staged local artifacts.

Exercise 8: add tool streaming safely

Define typed states for approval-required, tool progress, produced file, terminal success/failure, and timeout. Prove text cannot skip a pending approval and a named event cannot be silently discarded.

Exercise solutions

Solution 1

Maintain a bounded byte buffer until a complete SSE line delimiter appears, normalize CRLF, and accumulate fields until a blank line dispatches the event. UTF-8 may cross network chunks, so decode only complete buffered sequences. Property-test every split point and random partitions. The parser result and digest of original wire bytes must match.

Solution 2

Return a distinct enum carrying partial text, usage, and terminal reason. Do not emit a normal TextGenerationResponse. A retry is a new run with a new request ID; it may increase output budget only within context/cost policy and should reuse the same model/input identity.

Solution 3

Resolve authorization before publishing a tenant-scoped lease. Generate the server path from trusted digest/format, never from a caller path. Reference-count or lease staged artifacts by in-flight run; cleanup only expired zero-reference entries. Mount a tenant/run-specific view when stronger filesystem separation is required.

Solution 4

Accept only success status and text/event-stream for streaming. Require the echoed request ID to match the sent bounded ID. Cap header count/bytes. For errors, read at most a small fixed body, parse a known error envelope if possible, redact it, and categorize retryability without storing request content.

Solution 5

Embedded typically has zero network hops but one model copy per application process and shares its crash/rollout boundary. A service adds a hop and independent failure mode but can own one copy per device, batch across callers, and roll models separately. The correct choice follows measured latency, isolation, scaling, privacy, and availability constraints.

Solution 6

Use the same bundle except quantized weight representation, same request set, processor/template, backend, and limits. A candidate passes only if task/safety/calibration degradation is within the declared release budget and resource/SLO gains are material. Publish the candidate as a new exact model release.

Solution 7

Fetch in a network-isolated acquisition service with an allowlisted scheme, public-IP policy, redirect revalidation, blocked link-local/private/metadata ranges, DNS pinning policy, port rules, timeouts, byte limits while streaming, media sniffing, decoded bounds, and digest-addressed publish. The inference worker needs no general egress.

Solution 8

Use an enum whose transitions permit output only after the current tool/approval state resolves. Unknown event names fail under the pinned adapter. Approvals are authenticated actions bound to request/session/tool arguments and expire. Produced files enter the artifact-validation pipeline before display or later tool use.

Review checklist

You should now be able to answer:

Why is mistralrs serve -m org/model not a reproducible release?
Which identity does /v1/models prove, and which identities must deployment add?
Why is a health 200 insufficient for readiness?
Why must media staging precede the inference request?
Which URL forms can cause file-read or SSRF risk?
Why is content-part order semantic?
What success evidence beyond partial stream text is required?
Why does a closed client connection not prove device compute stopped?
How do max-seqs, batch size, and product admission differ?
What does paged attention improve, and what memory does it not remove?
Why can prefix caching cross a privacy boundary?
Why are --quant and an exact quantized artifact not the same identity?
When would you embed the Rust SDK instead of use a separate process?
Which mistral.rs features add execution authority and are disabled here?

If an answer is unclear, mutate the fixture and predict which validator must reject it.

References

What comes next

Candle exposed tensor/model mechanics. mistral.rs exposed a high-level multimodal generation service with batching and a streaming protocol. The next chapter integrates ONNX Runtime for portable specialist models—small classifiers, detectors, embedders, audio transforms, and other bounded graphs—then compares graph contracts, execution providers, preprocessing, dynamic shapes, threading, and output verification against the two paths built here.