Trait Objects and Runtime-Selected Providers

Generics choose concrete types at compile time. Production AI systems often choose a provider at runtime from configuration, tenancy, health, capability, cost, or an experiment assignment.

A trait object such as Arc<dyn Model> erases the concrete adapter type while retaining the Model contract. That flexibility is useful only when the erased trait remains honest and routing happens before an unsupported call.

Learning objectives

After this chapter, you will be able to:

• explain fat pointers, vtables, and dynamic dispatch conceptually;
• create &dyn Trait, Box<dyn Trait>, and Arc<dyn Trait> for different ownership needs;
• identify object-safety/dyn-compatibility constraints;
• compare static and dynamic dispatch using architectural requirements;
• design a typed runtime model registry;
• combine Arc<dyn Model> with a generic harness;
• preserve model identity and capability metadata through erasure;
• debug lifetime, Send, Sync, and missing-route failures;
• run a real runtime-selected companion model.

Dependencies and downstream use

This chapter depends on traits, generics, Arc, collections, async basics, and error types. It is a prerequisite for provider registries, tool plugins, grader collections, router cascades, and Mosaic’s runtime assembly.

configuration / router
        │ ModelId
        ▼
HashMap<ModelId, Arc<dyn Model>>
        │ selected concrete adapter erased behind trait
        ▼
Harness<Arc<dyn Model>, JsonContractVerifier, TraceSink>

Dynamic dispatch belongs at a real runtime choice. Inside a fixed tensor kernel or parser, generics or ordinary functions are often simpler.

What dyn Trait means

A trait object is a dynamically sized value representing some concrete type that implements a trait, but whose concrete identity is not statically named at the use site:

let model: Arc<dyn Model> = Arc::new(ScriptedModel::new(steps));

A pointer to a trait object is conceptually a fat pointer containing:

1. a pointer to concrete data;
2. a pointer to a vtable containing method implementations and layout/drop information.

Calling model.infer(...) performs dynamic dispatch through the vtable.

You normally use a trait object behind a pointer/reference because dyn Model itself is dynamically sized:

• &dyn Model borrows an implementation;
• Box<dyn Model> owns one implementation;
• Arc<dyn Model> shares ownership across tasks/registries;
• &mut dyn Model exclusively borrows a mutable implementation.

The pointer choice expresses ownership separately from dispatch.

Why a registry needs erasure

A HashMap has one value type. These adapters have different concrete types:

struct CloudTextModel { /* ... */ }
struct LocalMistralModel { /* ... */ }
struct ScriptedModel { /* ... */ }

Erasure makes the value type uniform:

struct ModelRegistry {
    models: HashMap<ModelId, Arc<dyn Model>>,
}

The registry can choose one adapter at runtime without an enum variant for every provider.

An enum is a strong alternative when the set is closed:

enum ConfiguredModel {
    Cloud(CloudTextModel),
    Local(LocalMistralModel),
}

Enum dispatch provides exhaustive matching and can expose variant-specific behavior. Trait objects support an open set of implementations and plugin-like assembly. Choose from whether the set is closed, not from performance folklore.

Object safety / dyn compatibility

Not every trait can become dyn Trait. The compiler needs to build a vtable whose method calls have known dispatch shape.

Common obstacles include:

• methods returning Self without Self: Sized;
• generic methods whose type is chosen per call;
• associated constants in unsupported object contexts;
• methods without an object-callable receiver;
• native async trait methods whose desugared return type is not object-safe under the selected design.

Example of a method that should be excluded from trait-object calls:

trait Model {
    fn infer(&self, request: ModelRequest) -> ModelFuture<'_>;

    fn configured(config: Config) -> Self
    where
        Self: Sized;
}

The constructor is available for concrete implementors but not through dyn Model.

The companion uses async-trait, which transforms async fn infer into a boxed future signature that can be called through a trait object. That choice is deliberate for runtime adapters.

Arc<dyn Model> and the generic harness

The generic harness requires M: Model. A trait object behind Arc is a distinct type; delegation connects it:

#[async_trait::async_trait]
impl<M> Model for Arc<M>
where
    M: Model + ?Sized,
{
    async fn infer(
        &self,
        request: ModelRequest,
    ) -> Result<ModelResponse, ModelError> {
        (**self).infer(request).await
    }
}

Now:

let selected: Arc<dyn Model> = registry.resolve(&model_id)?;
let harness = Harness::new(selected, verifier, trace);

works without changing the harness to dynamic dispatch internally. The assembly boundary erases only the model implementation.

?Sized permits M to be a dynamically sized trait object. Arc<M> remains a sized handle.

Ownership choices

Borrowed &dyn Trait

Use when the implementation already lives long enough and no independent ownership is needed:

fn health_check(model: &dyn ModelMetadata) -> Health;

Owned Box<dyn Trait>

Use for one owner:

struct ToolRegistration {
    tool: Box<dyn Tool>,
}

Moving the box moves the pointer, not the underlying allocation.

Shared Arc<dyn Trait>

Use when the registry and concurrent runs need independent handles:

let model = Arc::clone(registry.get(&id)?);

Arc does not make a non-thread-safe concrete adapter safe. The trait requires Send + Sync, so only compatible implementations can be registered. Runtime concurrency limits remain separate.

Do not use Arc<Mutex<dyn Model>> by default. It serializes calls and may hold a guard across await. If a local runtime is single-threaded, prefer one owner task receiving bounded requests.

Preserve identity after type erasure

Erasing the Rust concrete type must not erase release identity. Every response/trace should retain:

• registry ModelId;
• provider and provider-side model name;
• model artifact/version/revision;
• adapter version;
• tokenizer/chat-template identity;
• capability metadata;
• relevant decoding parameters;
• routing decision and alternatives.

The response’s self-reported model ID must be checked against the selected route. A provider alias can change under the same display name, so production release identity may need immutable provider revisions.

Do not use std::any::type_name as product identity. It is a Rust implementation detail, not a stable model release identifier.

Capability metadata before dispatch

The Model trait in Forge is intentionally narrow. Mosaic’s registry will pair the callable model with validated metadata:

struct RegisteredModel {
    id: ModelId,
    capabilities: ModelCapabilities,
    limits: ModelLimits,
    economics: CostProfile,
    model: Arc<dyn Model>,
}

The router checks:

• text/image/audio/video input support;
• structured-output support;
• context/output limits;
• residency/privacy policy;
• tool-call protocol;
• latency/cost class;
• health and concurrency capacity.

Only after eligibility is established does scoring select a route. An unsupported method should not be discovered by calling and receiving a runtime “no.”

Split capability traits further when invocation contracts differ. Metadata does not justify an everything-model trait.

Downcasting is usually a design smell

If code immediately asks which concrete type is behind dyn Model, the erased trait may be missing a capability:

// Smell:
if let Some(local) = model.as_any().downcast_ref::<LocalModel>() {
    local.special_reset();
}

Prefer:

• a separate lifecycle trait;
• explicit registry metadata;
• enum dispatch for a closed set;
• an adapter-managed behavior hidden behind the existing contract.

Downcasting can be appropriate in narrow infrastructure/testing contexts, but product policy should not depend on provider concrete types.

Trait-object lifetimes

This type:

Box<dyn Verifier + 'static>

means the object environment contains no non-static borrowed references. It can still be dropped when the box is dropped.

A borrowed object can carry a shorter lifetime:

fn temporary<'a>(config: &'a Config) -> Box<dyn Verifier + 'a>;

Registries usually need 'static implementations because they outlive request stacks. Put owned configuration or Arc<Config> inside adapters rather than borrowing startup locals.

Compiler defaults around omitted trait-object lifetimes can be subtle. Write explicit bounds in public/stored types when the ownership contract matters.

Error design at the registry boundary

Missing route is not provider failure:

enum RegistryError {
    Missing(ModelId),
    Ineligible {
        id: ModelId,
        missing: Capability,
    },
    Unhealthy(ModelId),
}

The router can react differently:

• Missing: configuration/program error or bad requested ID;
• Ineligible: select a compatible route;
• Unhealthy: choose fallback or defer;
• provider timeout after dispatch: retry/fallback under separate policy.

Do not return Option<Arc<dyn Model>> if the reason for absence drives policy.

Registry mutation also needs a duplicate policy. Silent replacement can switch production models without a trace. Prefer startup rejection or an explicit versioned update event.

Runtime routing without behavior drift

A route should produce a trace record:

task requirements
eligible candidates
policy exclusions
score inputs
selected model ID/release
fallback chain
budget assigned

Dynamic dispatch is mechanism; routing is product policy. Keep the policy testable with pure data where possible. A model registry must not select “first hash-map entry,” which is nondeterministic and semantically meaningless.

For experiments, assignment should be stable by subject/run under a recorded experiment version. For fallbacks, ensure provider errors do not cause an unbounded cascade.

Runnable companion implementation

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-harness --example runtime_model_registry
cargo test --workspace --examples

Expected output:

selected: small; model calls: 1

The implementation is crates/mosaic-harness/examples/runtime_model_registry.rs.

It:

1. registers two different scripted instances as Arc<dyn Model>;
2. resolves the runtime string "small" to a validated ModelId;
3. clones only the Arc handle;
4. passes the erased model through the real generic harness;
5. returns a typed missing-route error;
6. tests that missing selection fails before inference.

The concrete model type is erased, but response model ID, harness budgets, verification, and trace behavior remain.

Failure investigation: runtime route exists but is incompatible

Symptom: image task reaches a text-only adapter and fails after request construction.

Root cause: registry keyed only by ID and router skipped capability eligibility.

Investigation:

1. inspect task requirements and registry metadata;
2. verify metadata is versioned with the adapter/model release;
3. reproduce the eligibility filter without inference;
4. add a negative fixture for the missing capability;
5. ensure routing traces exclusions;
6. prevent construction of incompatible provider requests.

Fix: resolve RegisteredModel, validate capabilities/limits/policy, then obtain the callable model. A map hit alone is not eligibility.

Failure investigation: model cannot enter the registry

Symptom: compiler reports the concrete adapter is not Send or Sync.

Do not: add unsafe trait implementations or wrap everything in a mutex automatically.

Determine:

• is the SDK handle thread-confined?
• can each call create its own client?
• should one worker own the runtime and receive messages?
• is a non-thread-safe native context being shared?
• can the registry store a factory rather than a shared instance?

The type error is protecting the runtime’s threading invariant.

Common failure modes

Dynamic dispatch everywhere

Use it at runtime variability boundaries. Plain functions/generics stay clearer elsewhere.

Giant erased trait

Trait objects do not fix dishonest optional capabilities.

Missing release identity

The Rust vtable says nothing about which model artifact produced output.

Silent duplicate registration

Reject or explicitly version replacements.

Routing from hash-map iteration

Define stable eligibility, scoring, and tie-breaking.

Downcasting for product behavior

Add a capability or use an enum if variants truly matter.

Arc<Mutex<dyn Trait>> reflex

Choose an ownership/concurrency architecture based on the runtime’s actual constraints.

Security and performance implications

Security:

• registry entries are capabilities; restrict who can register/replace them;
• validate metadata rather than trusting adapter claims blindly;
• keep credentials inside adapters and out of Debug;
• ensure tenant/residency policy filters before dispatch;
• record immutable selected release identity;
• bound fallback chains and prevent policy downgrade.

Performance:

• one vtable call is negligible beside remote/local model inference in most application paths;
• boxed async futures allocate per call under async-trait;
• Arc reference counts add atomic operations;
• dynamic dispatch can inhibit inlining in tight CPU loops;
• registries simplify sharing expensive clients/runtimes but can retain them indefinitely.

Measure the hot path. Do not sacrifice replaceability at an inference boundary to save nanoseconds you have not observed.

Exercises

Recall

1. Why is dyn Trait used behind a pointer?
2. What is conceptually stored in a trait-object fat pointer?
3. How do Box and Arc change ownership but not the trait contract?
4. Why can a generic method make a trait non-dyn-compatible?

Implementation

1. Reject duplicate model registration with a typed error.
2. Add capability metadata and an eligibility check before resolve.
3. Add a stable fallback list with maximum depth.
4. Implement a registry of Arc<dyn Verifier> with version IDs.
5. Add a factory trait for adapters that must create one client per run.

Design

Compare enum, generic, and trait-object dispatch for:

• three built-in output validators;
• arbitrary provider adapters loaded at startup;
• tensor kernels in one local runtime;
• a runtime tool registry;
• two storage backends selected at compile time;
• grader plugins selected by eval configuration.

State openness, ownership, object safety, metadata, and performance needs.

Failure lab

Test:

• missing model ID;
• duplicate registration;
• incompatible modality;
• equal routing scores;
• unhealthy primary with bounded fallback;
• fallback that would violate residency policy;
• provider-reported model ID differing from selected ID.

Solution guidance

Registration should use the map entry API and return the attempted ID without replacing the old model.

Eligibility should produce explicit exclusion reasons. Keep selection pure over metadata so unit tests do not call providers.

Fallback is an ordered, bounded policy artifact. Re-run eligibility for every fallback; health failure must not bypass privacy or authority constraints.

A factory can return Arc<dyn Model> or an owned Box<dyn Model> depending on whether a created adapter is shared. It must define initialization failure and secret handling.

Check your understanding

• Does Arc<dyn Model> mean the inner model is mutable?
• Can a non-Sync adapter be safely registered under Model: Sync?
• Why does the companion implement Model for Arc<M>?
• What does erasure remove, and what identity must traces retain?
• When is enum dispatch a better fit than trait objects?
• Why should missing ID and ineligible capability be different errors?
• Does dynamic dispatch define routing policy?

Completion checklist

• I use trait objects only where runtime type choice is real.
• I select the correct ownership pointer.
• I design traits for dyn compatibility intentionally.
• I retain model, adapter, tokenizer, and policy release identity.
• I validate capabilities before dispatch.
• I define duplicate, missing, unhealthy, and fallback behavior.
• I do not bypass thread-safety with unsafe implementations.
• I ran the runtime registry and its missing-route test.

Authoritative references

• The Rust Programming Language: Trait Objects
• The Rust Reference: Trait objects
• The Rust Reference: Dyn compatibility
• std::sync::Arc
• std::boxed::Box
• async-trait documentation