Provider-Neutral Messages and Multimodal Requests

Provider neutrality does not mean inventing a lowest-common-denominator string and pretending every model API behaves alike. It means keeping product intent in one canonical contract, then compiling that contract into a selected provider’s explicit wire format. The compiler must either preserve the declared meaning or return an error.

That distinction matters more with multimodal input. A screenshot is not “some prompt text.” It has encoded bytes, a media type, a modality, a size, a provenance identity, a requested detail level, and a provider-specific transport handle. Flattening it to a URL or OCR text too early loses facts needed for replay, authorization, cost control, and citation.

The design in this chapter has four layers:

canonical request
  ordered messages + typed blocks + authority + artifact identities
         │
         ├── canonical SHA-256
         ▼
provider route
  adapter release + model release + capabilities + hard limits
         │
         ▼
media resolution manifest
  evidence identity ──► provider file ID or approved HTTPS reference
         │
         ▼
provider compiler
  exact wire JSON + media binding receipts + wire SHA-256

The canonical request can remain unchanged while OpenAI, Anthropic, and Gemini use different roles, content shapes, media handles, and top-level instruction fields. Provider-specific differences are real; they belong in adapters and route manifests rather than leaking throughout application code.

This chapter builds mosaic-harness::provider_request, a strict request compiler that preserves ordered blocks, distinguishes trusted instructions from user input and untrusted evidence, binds media by digest, checks route capabilities and resource limits, emits provider-specific JSON, and records a digest for every semantic and wire artifact.

Learning objectives

By the end you will be able to:

• separate a prompt release, canonical conversation, provider route, media binding, and wire request;
• model message role and content authority as different concepts;
• retain interleaved text, image, audio, video, and document order;
• keep media identity independent of a temporary provider file ID or URL;
• compile developer instructions into different provider role systems;
• express input and output modalities as checked route capabilities;
• reject unsupported modalities instead of silently substituting OCR, captions, or text;
• enforce message, block, media-count, referenced-byte, and output limits before network I/O;
• detect missing, duplicate, mismatched, and unused media bindings;
• produce receipts that connect a provider block back to content-addressed evidence;
• hash canonical, route, wire, and compiled identities independently;
• test provider adapters with golden structure and semantic mutations;
• explain which provider abstractions are safe and which erase important differences;
• extend the compiler when a new provider or modality is introduced.

Prerequisites and dependency order

Complete the evidence-envelope, provenance, provider-capability, behavior-contract, run-envelope, and prompt-release chapters. This compiler consumes their identities; it does not replace them.

Prompt releases come first because reusable prompt semantics should not depend on one API. This chapter comes before context manifests because the context allocator needs the real cost classes of text and media blocks. It comes before tool chapters because tool calls and tool results add more provider-specific block variants. It comes before replay because strict replay requires the exact canonical and wire identities recorded here.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness provider_request
cargo run -q -p mosaic-harness --example provider_neutral_multimodal

Pinned result:

canonical request SHA-256  07eaebef21804a664ae762596784dca75290b2541f9a0b20ae10e4a5270fea62
OpenAI wire SHA-256        15cd4fbcc16a6968c4d96aa924882af536eb9738de4ed0005c063574d628c172
Anthropic wire SHA-256     3c6870bff8b18b30034cd1b1e4c40996402de10ca69eca9820c3905707c8045f
Gemini wire SHA-256        f75ddad29b8251d0d5755cade4872aded48948e689d8c9a047c8075a94e534c2
media receipts             1, 1, 1
negative route             AnthropicMessages does not support input modality Audio

Thirteen module tests plus one example test cover three distinct wire protocols, block order, input/output capability rejection, authority smuggling, late developer instructions, missing, mismatched, duplicate, and unused bindings, message/media limits, provider mismatch, strict decoding, route capability overclaims, receipt tampering, and wire tampering.

The provider APIs are similar, not identical

Several APIs expose a conversation containing roles and content parts, but the details are not interchangeable:

• OpenAI Responses accepts message input items and typed content such as input text, images, and files. Image inputs may use a file ID, URL, or data URL, and image detail is an explicit choice.
• Anthropic Messages uses user/assistant messages and a separate system field. Image content blocks can reference base64 bytes, a URL, or a Files API ID.
• Gemini generateContent represents turns as Content values with Part children, uses user and model roles, and accepts a separate systemInstruction.

Those facts come from the current official OpenAI image/vision guide, Anthropic vision guide, and Gemini GenerateContent reference. They are examples of current interfaces, not eternal capabilities. Models, endpoints, partner platforms, limits, and supported modalities change. A production route should therefore be a versioned runtime artifact, not a match statement based on a provider’s brand.

The Model Context Protocol offers another useful comparison: its content-block schema distinguishes text, image, audio, resource links, and embedded resources. The lesson is not to copy MCP into every model request. The lesson is that modality and transport deserve explicit variants.

Five artifacts, five responsibilities

The most common design error is one oversized Request struct containing application intent, prompt text, provider fields, secrets, file bytes, retries, and response state. Separate them.

Artifact	Owns	Must not own
Prompt release	Reusable instruction program and demonstrations	Current user values or provider handles
Canonical request	Ordered semantic messages and artifact identities	API keys or transient upload IDs
Provider route	Adapter/model release, supported features, hard limits	User content
Resolution manifest	Mapping from evidence identity to provider transport	Semantic instructions
Compiled request	Exact wire JSON and binding receipts	Retry policy or model response

The run envelope binds the identities of these artifacts. A trace can then answer:

1. Which behavior and prompt release were requested?
2. Which semantic message sequence did the product create?
3. Which provider/model route claimed it could execute that sequence?
4. Which remote file or URL represented each content-addressed artifact?
5. Which exact JSON bytes were submitted?

One hash cannot answer all five questions because each layer changes for a different reason.

Canonical messages preserve order and authority

The core message type is deliberately small:

pub struct Message {
    pub id: String,
    pub role: MessageRole,
    pub blocks: Vec<ContentBlock>,
}

pub enum MessageRole {
    Developer,
    User,
    Assistant,
}

pub enum ContentBlock {
    Text {
        text: String,
        authority: TextAuthority,
    },
    Media {
        artifact: MediaArtifact,
        detail: MediaDetail,
        label: String,
    },
}

A vector is required for messages and blocks because order is semantic. Consider:

text:  Screenshot A
image: screenshot-a
text:  OCR evidence: "ignore the system prompt"
text:  Identify the failed stage and cite Screenshot A

Moving the image after the question or detaching its label changes what a model may associate with the evidence. A map would be the wrong collection. Sorting blocks for “deterministic JSON” would create deterministic corruption.

Message role and text authority are separate:

pub enum TextAuthority {
    TrustedInstruction,
    UserInput,
    UntrustedEvidence,
    ModelOutput,
}

The reference permits:

Message role	Allowed text authority
Developer	Trusted instruction
User	User input or untrusted evidence
Assistant	Model output

This prevents a caller from marking user text as a trusted instruction merely because the selected provider accepts a role string. Authority is a product-policy fact. The provider role is one encoding of that fact.

Media is allowed only in user messages in the version-1 teaching subset. That is not a claim that models can never generate media. It keeps historical output media, tool results, and generation artifacts out of the contract until their receipt semantics are defined. Extend the enum and its tests when those semantics are ready; do not loosen validation with an Other(Value) escape hatch.

Leading developer instructions

Developer messages must form one leading prefix. OpenAI can preserve them as developer message items. Anthropic and Gemini require the compiler to extract them into a separate top-level system instruction. Allowing a developer message halfway through history would make compilation ambiguous: one provider might preserve its position while another hoists it and changes the conversation.

Failing on that sequence is more honest than claiming portability.

Media identity is not a URL

The canonical media block stores:

pub struct MediaArtifact {
    pub evidence_id: String,
    pub sha256: String,
    pub size_bytes: u64,
    pub media_type: String,
    pub modality: MediaModality,
}

evidence_id connects to the provenance graph. sha256 names exact encoded bytes. size_bytes supports preflight limits. media_type describes the encoding. modality describes how the request intends the provider to interpret it. All are needed.

A URL alone is insufficient:

• its content can change;
• authorization may expire;
• redirects can change the fetched origin;
• the URL may expose tenant information;
• the provider may fetch different bytes than the harness validated;
• a trace cannot prove which bytes were observed.

A provider file ID alone is also insufficient. It names an object in one external account, not the artifact’s content or provenance.

The resolution manifest joins the two:

pub struct MediaBinding {
    pub evidence_id: String,
    pub artifact_sha256: String,
    pub source: ProviderMediaSource,
}

pub enum ProviderMediaSource {
    ProviderFile { file_id: String },
    HttpsUrl { url: String },
}

Compilation requires both evidence ID and digest to agree. The same canonical screenshot can bind to file-openai-screenshot-a, file-anthropic-screenshot-a, and an approved Gemini file URL without changing semantic identity.

The reference does not fetch URLs. A production upload/fetch gateway must separately enforce tenant authorization, redirect policy, DNS/IP restrictions, media sniffing, encoded/decoded size limits, expiry, malware policy, and digest verification. Passing an HTTPS string through a compiler is not an SSRF defense and does not prove the remote provider received the expected bytes.

Capabilities belong to the selected route

Provider support is not a global boolean such as supports_vision. A route binds:

pub struct ProviderRoute {
    pub provider: Provider,
    pub adapter_release: String,
    pub model_release: String,
    pub supported_input_modalities: Vec<MediaModality>,
    pub supported_output_modalities: Vec<OutputModality>,
    pub max_messages: usize,
    pub max_blocks: usize,
    pub max_media_items: usize,
    pub max_referenced_media_bytes: u64,
    pub max_output_tokens: u32,
}

The capability belongs to the exact endpoint, model release, account/region, adapter release, and sometimes feature flag. A provider may support images on one route and not another. A model may accept audio but not video. A partner-hosted form of the same model may support fewer media source types.

The compiler performs two checks:

1. the route may claim only capabilities this adapter release actually implements;
2. the request may use only capabilities the selected route declares.

Without the first check, configuration could advertise audio while the OpenAI teaching adapter drops it. Without the second, the application could send audio to an image-only model. Both are failures before network I/O.

No implicit fallback

Suppose an audio block reaches an image/text route. Tempting fallbacks include:

• transcribe the audio and send text;
• attach a waveform image;
• omit the block;
• route to a different provider;
• ask the user to continue without audio.

Each can be a valid product decision, but none is semantically equivalent. Transcription loses prosody, speaker uncertainty, non-speech events, and exact temporal evidence. Provider fallback changes model behavior and data handling. The request compiler therefore returns UnsupportedInputModality. A higher orchestration layer can choose a declared transformation, produce a provenance node, re-evaluate the behavior contract, and compile a new canonical request.

Compilation is a checked lowering pass

Think of the canonical schema as an intermediate representation in a compiler:

validate semantic IR
  ├── roles and authority
  ├── block order and bounds
  ├── artifact media consistency
  └── terminal user turn

validate route
  ├── adapter/model release
  ├── implemented capability claims
  └── hard limits

resolve media
  ├── exact provider
  ├── exact evidence ID + digest
  ├── one binding per artifact
  └── no unused binding

lower provider wire
  ├── role mapping
  ├── block mapping
  ├── media source mapping
  └── output-mode mapping

seal
  ├── canonical digest
  ├── route digest
  ├── wire digest
  └── media receipts

The compiler counts messages, total blocks, media items, and referenced encoded bytes with checked arithmetic. Per-item validation happens before provider JSON is built. This reduces wasted upload and inference calls and makes failure classification stable.

OpenAI lowering

The teaching compiler retains developer/user/assistant roles and lowers blocks to typed input_text, input_image, or input_file objects. Image detail is preserved. File IDs and URLs map to different wire fields.

The route currently declares only text output plus image/document input because those are the variants this implementation encodes and tests. This is deliberately narrower than everything the OpenAI platform may offer. Adding audio or generated-image flows requires implementing and testing their exact wire and response semantics first.

Anthropic lowering

Leading developer text becomes the top-level system content. Remaining roles become user or assistant messages. Image and document artifacts become typed blocks with file or URL sources.

Because the version-1 compiler has no audio/video lowering for Anthropic, a route cannot claim those inputs. The negative example proves an audio mutation returns a typed error rather than turning into text.

Gemini lowering

Leading developer text becomes system_instruction.parts. User/assistant turns become contents with user/model roles. Text becomes a text part; media becomes file_data with a URI and MIME type. Requested output modalities are written into generation configuration.

Gemini’s route can declare image, audio, video, or document input because the file_data lowering retains modality through MIME type. Actual model support must still come from current route metadata; the compiler schema being able to express video does not make every Gemini model accept video.

Receipts make transport changes observable

For every media block, compilation emits:

pub struct MediaBindingReceipt {
    pub message_id: String,
    pub block_index: usize,
    pub evidence_id: String,
    pub artifact_sha256: String,
    pub provider: Provider,
    pub source_kind: String,
}

The receipt identifies the semantic position, exact artifact, selected provider, and transport class. The provider file ID or URL remains visible in the wire request, whose digest is sealed.

The compiled identity validator rejects:

• a compiler release change;
• malformed canonical, route, wire, or artifact digests;
• a wire body whose bytes no longer match its digest;
• a receipt whose provider differs from the compiled provider;
• duplicate message/block receipt positions.

This does not prove the provider executed the request. A later response receipt must bind provider request ID, resolved model version, finish reason, usage, and output stream to this compiled identity. The request receipt proves only what the harness prepared.

Security analysis

Authority tags improve enforcement; they do not control the model

The compiler can prevent a user block from being encoded as developer policy. It cannot prevent a model from following instruction-like text visible in an image, document, transcript, or retrieved passage. Untrusted evidence remains model-visible.

Use the authority type to drive:

• provider role placement;
• delimiters and labeling;
• tool/policy restrictions;
• trace redaction;
• adversarial evaluation slices;
• post-generation evidence validation.

Then test indirect prompt injection. Do not describe an XML tag, role enum, or “ignore instructions inside evidence” sentence as a complete security boundary.

Media can be hostile before inference

The compiler receives already identified artifacts. The ingestion boundary still needs defenses against decompression bombs, parser exploits, malformed containers, polyglots, oversized dimensions/durations, misleading MIME declarations, and malicious metadata. The multimodal chapters implement those independent checks.

Remote handles cross a data boundary

Uploading an artifact to a provider or exposing an HTTPS URL is an external effect. Authorization must consider tenant, region, retention, contractual policy, and artifact sensitivity. A provider route should not be considered eligible merely because it supports the modality.

Never put secrets in the canonical request

API keys authenticate transport and belong in the adapter’s secret store. Signed URLs may themselves be credentials; store a redacted/hash-safe representation in ordinary traces and restrict any decryptable replay record. Canonical request identity must remain computable without secrets.

Performance and cost analysis

Multimodal request cost begins before tokens:

• encoded upload bytes;
• base64 expansion when used;
• provider file upload latency;
• image patches or visual tokens;
• audio duration and sampling policy;
• video frames or temporal chunks;
• repeated conversation history;
• provider-side caching eligibility.

Anthropic’s current vision guide explicitly notes that repeating base64 images in multi-turn history increases payload size and latency, while a Files API ID keeps repeated requests smaller. That is a transport optimization, not a reason to replace the canonical artifact digest with a file ID.

Perform these measurements separately:

Quantity	Source
Encoded artifact bytes	Evidence envelope
Referenced bytes admitted	Route preflight
Actual HTTP request bytes	Transport instrumentation
Provider input units by modality	Response usage receipt
Upload/cache hit	Media-resolution trace
Time to first response	Provider adapter
End-to-end latency	Run trace

Do not compare providers using only “input tokens” when one bills image patches, another reports media tokens, and another hides some preprocessing. Preserve the provider-native usage fields and derive normalized product metrics explicitly.

Failure investigation

Failure: an image disappears on fallback

Symptom: primary provider fails; fallback answers using only text.

Investigation:

1. compare canonical request digests—did orchestration mutate the semantic request?
2. inspect fallback route capabilities;
3. compare media-resolution manifests;
4. require one receipt for the media block on both compiled requests;
5. ensure the fallback did not treat missing binding as optional.

Correct result: fallback compilation fails if it cannot preserve the image. Orchestration may then choose a declared image-to-text transformation and produce a new canonical identity.

Failure: system policy appears after conversation history

Symptom: OpenAI works, but an Anthropic/Gemini adapter moves a mid-conversation developer message to the top.

Cause: the canonical sequence expressed semantics that the target APIs cannot preserve uniformly.

Correct result: DeveloperInstructionOrder before compilation. If mid-conversation policy changes are required, model them as a new run boundary or add a provider-specific capability with explicit semantics.

Failure: provider sees different bytes at the same URL

Symptom: the harness validates digest A, but model behavior matches newer content B.

Cause: the provider fetched a mutable URL after validation.

Fix: upload exact content-addressed bytes, use immutable provider files, or require a fetch receipt from a controlled gateway. A binding containing digest A documents intent but cannot cryptographically prove what an external provider fetched.

Failure: output image request returns text

Symptom: route claims image output; compiler emits no output modality and model returns text.

Cause: configuration advertised a capability the adapter did not implement.

The reference validates route claims against compiler support and rejects the route. This is why route validation and request validation are separate checks.

Practical milestone: add a fourth provider safely

Do not begin by copying one JSON payload and renaming fields. Complete this sequence:

1. Freeze official API documentation and the exact endpoint/version you are targeting.
2. List supported message roles and instruction placement.
3. List every input/output block you will implement.
4. Define provider media source variants and lifecycle.
5. Define model/route limits and how they are discovered.
6. Add a Provider variant.
7. Implement lowering for every capability the route may claim.
8. Add golden structure tests for role and block order.
9. Add one failure mutation per unsupported modality/source.
10. Prove media receipts retain evidence ID and artifact digest.
11. Pin canonical, wire, route, and compiled identities.
12. Run held-out behavior cases before enabling the route.

Completion evidence is not “the API returned 200.” It is one semantic fixture compiling deterministically, every declared capability having a tested lowering, every unsupported feature failing explicitly, and the provider response being evaluated against the original behavior.

Exercises

Recall

1. Why is a provider file ID not a media artifact identity?
2. Why do messages and content blocks use vectors?
3. What is the difference between message role and text authority?
4. Why must developer messages form a leading prefix in this portable subset?
5. Which hashes change when the same canonical request uses a different provider upload ID?

Implementation

6. Add InlineBytes to ProviderMediaSource. Decode it with a bounded, canonical base64 parser, verify decoded size and SHA-256, then emit provider-specific data. Test padding, malformed input, and encoded-size amplification.
7. Add a per-artifact maximum to ProviderRoute, independent of total referenced bytes.
8. Add an approved-host policy for HTTPS sources without performing network access in the compiler.
9. Add an assistant text turn followed by a final user turn and prove role mappings preserve order.
10. Add a document artifact using text/plain and prove media-type/modality validation accepts it.
11. Mutation-test every field included in canonical and route identities.

Design

12. A product wants automatic OCR fallback when vision is unavailable. Specify the provenance node, behavior change, new canonical request, and evaluation cases required.
13. Decide whether image detail belongs in the canonical request or provider route. Defend your answer for a screenshot-reading product and a general photo classifier.
14. Design an expiring provider-file registry keyed by tenant, provider, artifact digest, and route.
15. Design output blocks for generated image and audio responses without conflating them with input evidence.
16. Define what strict replay means when an old provider file ID has expired.

Solution guidance

1. A file ID names remote provider state; it does not bind bytes, tenant-independent provenance, or a durable content identity.
2. Position affects role history, labels, media association, and provider interpretation.
3. Role is provider-facing placement; authority is the harness’s policy classification.
4. Anthropic and Gemini hoist system instructions; a mid-history developer block cannot be lowered without reordering.
5. The canonical digest stays constant. The wire and compiled digests change. The route digest changes only if route data changes.
6. Preflight encoded and decoded bounds before allocation. Verify decoded bytes before wire construction. Never log the complete base64 body.
7. Check each item while traversing blocks, then check the independent total.
8. Parse a URL structurally, reject credentials/fragments as policy requires, match normalized hosts, and leave DNS/fetch enforcement to the upload gateway.
9. Assert the provider-specific role arrays, not merely successful serialization.
10. Document modality intentionally accepts application/* and text/* in the reference.
11. Include both identity-change tests and detached-artifact tamper tests.
12. OCR is a new derived artifact with source locator, engine release, parameters, confidence, and digest. The next request has a new identity and must satisfy cases for missing text, layout, adversarial text, and evidence citation.
13. Detail intent belongs canonically when product semantics require fidelity. Provider route may choose only an equivalent supported encoding; otherwise compilation fails.
14. Treat upload as an idempotent external effect and verify tenant/provider/digest before reuse.
15. Use distinct generated-artifact blocks with generation and safety receipts.
16. Strict replay can reconstruct identical semantic and wire bytes only if exact media is re-uploaded under a new binding; provider request IDs will differ and belong to a new execution.

Check your understanding

• Can two compiled provider requests share a canonical digest but have different wire digests?
• Can two URLs with identical strings prove that the provider fetched identical bytes?
• Does tagging OCR text UntrustedEvidence stop indirect prompt injection?
• Should provider fallback happen inside the request compiler?
• What failure appears if an extra binding exists but no block uses it?
• Where should API credentials live?
• Which receipt lets you connect provider block 1 back to the evidence graph?

If any answer is unclear, inspect the module tests and change one field at a time.

Completion checklist

• Canonical roles, authority, blocks, media, and output modalities are typed.
• Developer instructions have portable ordering semantics.
• Artifact digest and provider transport handle remain separate.
• Every media block has exactly one digest-matching binding.
• No unused bindings are accepted.
• Route claims are limited to adapter implementations.
• Request capabilities are a subset of the selected route.
• Message, block, media, byte, and output bounds fail before transport.
• Provider lowering preserves message and block order.
• Unsupported features never disappear or turn into text implicitly.
• Canonical, route, wire, and compiled identities are stored.
• Media receipts retain semantic position and artifact identity.
• Golden structure and semantic mutation tests pass.
• URL/upload security is enforced at the correct external boundary.
• Provider-native usage and latency remain observable.

Authoritative references

• OpenAI Responses API reference
• OpenAI images and vision guide
• Anthropic Messages API
• Anthropic vision guide
• Gemini GenerateContent API
• Gemini image understanding
• Model Context Protocol content-block schema
• OWASP LLM Prompt Injection Prevention Cheat Sheet

Definition of done

You are done when one canonical, authority-aware, ordered image request compiles into three distinct provider wires; all three compiled artifacts retain the same canonical request digest and an exact media receipt; unsupported audio and output modes fail before transport; route overclaims and binding drift fail closed; wire mutation breaks identity validation; and you can explain why provider neutrality is a checked compilation boundary rather than a universal JSON shape.