Day 25 — Run a Local Model Boundary

Today the transparent block becomes a deployable inference boundary.

Inspect before loading

cd rust/rust-ai-engineering
cargo run -q -p mosaic-models --example inspect_model_bundle
cargo run -q -p mosaic-ml --example candle_model_pipeline
cargo run -q -p mosaic-ml --example mistral_multimodal_service

A model bundle includes more than weights: architecture/config, tokenizer and special tokens, tensor names and dtypes, generation defaults, provenance, license, and release identity. Validate compatibility before allocating device memory.

Keep topology behind one capability

pub trait TextGenerator: Send + Sync {
    async fn generate(&self, request: GenerationRequest)
        -> Result<GenerationResponse, InferenceError>;
}

Implementations may embed Candle in-process, call a local mistral.rs server, or reach a remote provider. The domain request does not expose vendor wire objects.

Measure the real constraints

Record weight bytes, runtime overhead, KV-cache estimate, batch size, context length, device, quantization, prefill time, decode tokens/second, and peak memory. An out-of-memory event is often a capacity-planning failure before it is a model failure.

Blocking accelerator work must not occupy an async executor thread indefinitely. Use a bounded blocking pool or inference service, admission control, and cancellation semantics that match the runtime.

Break it deliberately

• wrong tokenizer for the weights;
• missing tensor;
• unsupported dtype/device pair;
• context that fits advertised length but exceeds memory;
• unbounded concurrent generations;
• service claims ready before weights load.

Completion proof

Run the same typed request through a scripted adapter and one local topology. Compare behavior, latency, memory, and failure reporting without changing the caller.

Deep references: Model artifacts, Quantization, memory, batching, and device, and Candle integration.

Next: Day 26 — Route Small Models →.