One Evidence Envelope Without Erasing Modality

A transcript is not audio. OCR text is not an image. Sampled frames are not a video. A caption is not the pixels that caused it.

The first multimodal design problem is therefore not “How do I send four files to a model?” It is:

How can one system identify, store, authorize, trace, retrieve, and cite mixed evidence without flattening away the coordinates and physical facts needed to interpret each modality?

The answer is one envelope with common operational fields and one tagged body with modality-specific fields. Common does not mean generic JSON. It means the fields that are genuinely shared—identity, encoded-byte reference, acquisition class, optional acquisition time, and schema version—have one contract. Text, image, audio, and video remain different Rust enum variants.

This chapter implements the first layer in a new mosaic-evidence crate. It parses a four-record JSONL fixture under strict size/schema rules and proves that the body cannot silently accept fields from the wrong modality. It records declared acquisition metadata, never mistakes it for verified decode results, and keeps raw media bytes outside the JSON manifest.

Learning objectives

By the end you will be able to:

distinguish encoded media, an evidence envelope, derived artifacts, observations, and claims;
identify the small set of fields truly common to text, image, audio, and video;
use a tagged Rust enum instead of an untyped property bag;
explain why one envelope does not imply one processing path;
separate declared metadata from decoded/verified metadata;
preserve original encoded bytes by content reference instead of embedding base64 in JSON;
state what a SHA-256 reference proves and does not prove;
validate bounded evidence IDs, digests, sizes, and media types;
enforce a media-type/modality consistency rule;
design strict Serde representations that reject unknown fields;
version the manifest and reject unsupported future versions;
bound a JSON record before deserializing it;
use exhaustive matching to route modality-specific work;
explain why acquisition time is not audio/video timeline time;
identify missing provenance/locator/trust information that later chapters must add;
build ordinary, boundary, and hostile-input fixtures for the envelope;
extend the schema without converting it into “anything goes.”

Prerequisites and dependency order

Part 3 established model/runtime capability boundaries. This part changes the thing that enters the harness: not one prompt string, but a set of typed evidence artifacts.

Read these foundations first:

Strings, bytes, paths, and Unicode distinguishes text from arbitrary bytes and validates boundaries before decoding.
Serde, derive macros, and generated contracts explains why deserialization must not bypass domain validation.
Content-addressed artifacts stores and verifies bounded bytes.
Provider-neutral capability traits keeps operations honest; this part will later add image/audio/video capabilities separately.

This chapter handles acquisition-envelope shape. Chapter 2 adds provenance graphs, locators, trust, and transformations. Chapters 3–6 decode and normalize each modality. Do not treat the minimal declared detail fields here as final media metadata.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example evidence_envelopes

The example parses:

fixtures/evidence/envelopes.jsonl

It returns four records:

Evidence ID	Media type	Typed modality facts
`incident-note`	`text/plain`	declared UTF-8, four lines
`dashboard-shot`	`image/png`	declared 1440 × 900 pixels
`operator-note`	`audio/wav`	declared 3 s, 48 kHz, one channel
`screen-recording`	`video/mp4`	declared 12 s, 1920 × 1080, one video and one audio track

Every record also contains:

schema version
stable evidence ID
SHA-256-shaped content reference
encoded byte length
media type
source kind
optional acquisition timestamp

The repeated aaaa…/bbbb… digests are visibly synthetic fixture values. The type validates their shape. A real acquisition path must compute the digest from the bytes and verify it again on read; it must not trust a digest supplied by a user.

A vocabulary that prevents category mistakes

Encoded artifact

The original byte sequence: UTF-8 file, PNG, WAV, MP4, or another registered format. It has a length and cryptographic digest. Those bytes can be decoded differently by different implementations, so preserve them.

Evidence envelope

A small manifest record that identifies the encoded artifact and stores acquisition-level facts needed to choose the next validator/decoder. It is not the media itself.

Decoded representation

Pixels with orientation/color interpretation, PCM samples with a sample clock, video packets/frames with time bases, or Unicode text with offset policy. Decoding can fail or disagree with declared metadata.

Derived artifact

Something produced from evidence: normalized text, crop, OCR spans, resampled audio, transcript, frame, shot list, embedding, caption, or generated media. It needs a parent and transform identity. That lineage is Chapter 2.

Observation

A detector/model/human output with method, confidence/uncertainty, and locator: “OCR found 500 in this polygon” or “speech recognizer proposed these words from 2.1–3.7 seconds.”

Claim

A statement the product may present: “the deploy completed before the error spike.” Claims cite evidence/observations; neither a high similarity score nor a caption is automatically proof.

Flattening these into one content: String field destroys the distinctions the harness needs to verify and cite.

What genuinely belongs in the common envelope?

The runnable type is:

pub struct EvidenceEnvelope {
    pub schema_version: u16,
    pub evidence_id: EvidenceId,
    pub artifact: EncodedArtifact,
    pub source_kind: SourceKind,
    pub acquired_at_unix_ms: Option<u64>,
    pub body: EvidenceBody,
}

`schema_version`

Defines how to interpret the whole record. Version 1 accepts exactly version 1. An unknown future version is not “close enough”; accepting it while ignoring new fields could invert meaning or trust.

`evidence_id`

A bounded application identity for references within a manifest/run. It is not the content digest. Two evidence items can reference the same bytes but have different acquisition/authorization contexts. Conversely, one logical item can gain derived children with new digests.

The current ID permits 1–128 ASCII letters, digits, hyphens, and underscores. It is safe for logs/keys but is not globally unique by itself; scope it by tenant/collection/run in storage.

`artifact`

pub struct EncodedArtifact {
    pub sha256: Sha256Digest,
    pub size_bytes: u64,
    pub media_type: MediaType,
}

It references encoded bytes. size_bytes must be nonzero. The digest is exactly 64 lowercase hexadecimal characters. A real store retrieves under a bounded size and recomputes the digest.

`source_kind`

Version 1 has:

user_upload
file_import
url_fetch
device_capture
system_record

This is deliberately coarse. It can drive initial policy without placing an arbitrary URL or local path in the core envelope. Chapter 2 adds a protected source locator, acquisition receipt, actor, trust assertions, and transformations.

`acquired_at_unix_ms`

When the system acquired/captured the evidence, when known. It is optional rather than invented. It is not:

filesystem modification time;
camera creation metadata;
the start of an audio/video internal timeline;
proof that a device clock was accurate;
a timezone-bearing human timestamp.

Later schemas attach clock source and uncertainty. Keep one numeric instant only when its semantics are actually known.

`body`

The tagged modality data. This is the key to avoiding erasure.

Use a tagged enum, not a bag of nullable fields

The body is:

#[serde(
    tag = "modality",
    content = "details",
    rename_all = "snake_case",
    deny_unknown_fields
)]
pub enum EvidenceBody {
    Text(TextDetails),
    Image(ImageDetails),
    Audio(AudioDetails),
    Video(VideoDetails),
}

JSON becomes:

{
  "modality": "audio",
  "details": {
    "declared_duration_ms": 3000,
    "declared_sample_rate_hz": 48000,
    "declared_channels": 1
  }
}

Serde’s enum representation guide calls this an adjacently tagged representation: the discriminator and variant content are adjacent. It has three advantages here:

a reader sees the modality explicitly;
Rust matching is exhaustive;
deny_unknown_fields rejects image width accidentally placed in text details.

Compare with:

struct ErasedMedia {
    text: Option<String>,
    width: Option<u32>,
    sample_rate: Option<u32>,
    duration: Option<u64>,
    extras: HashMap<String, Value>,
}

That structure permits nonsense: text plus sample rate, image without height, video represented by only a duration, and unknown fields silently deciding behavior. Validation becomes a growing web of boolean implications. An enum makes the top-level state valid by construction.

The four version-1 bodies

Text

pub struct TextDetails {
    pub declared_charset: Option<String>,
    pub declared_line_count: Option<u64>,
}

Both are declarations from acquisition metadata. The parser has not yet decoded bytes or verified UTF-8/line policy. Charset is a bounded printable ASCII label. A present line count must be nonzero. Chapter 3 will define Unicode normalization and source-coordinate stability.

Image

pub struct ImageDetails {
    pub declared_width_px: Option<u32>,
    pub declared_height_px: Option<u32>,
}

Dimensions are both present and nonzero or both absent. A single width is not useful geometry. These are encoded/container claims; Chapter 4 applies orientation, decode limits, color policy, and verified pixel dimensions.

Audio

pub struct AudioDetails {
    pub declared_duration_ms: Option<u64>,
    pub declared_sample_rate_hz: Option<u32>,
    pub declared_channels: Option<u16>,
}

Every present value is nonzero. The fields do not define codec, sample format, channel layout, time base, priming/padding, or decoded sample count. Chapter 5 does.

Video

pub struct VideoDetails {
    pub declared_duration_ms: Option<u64>,
    pub declared_width_px: Option<u32>,
    pub declared_height_px: Option<u32>,
    pub declared_video_tracks: Option<u16>,
    pub declared_audio_tracks: Option<u16>,
}

Video commonly contains synchronized audio. video/mp4 therefore remains a video artifact while the detail record can declare audio tracks. RFC 6838 defines the video top-level type broadly and explicitly recognizes coordinated sound in video formats.

A zero audio-track count is meaningful: video can be silent. A declared video-track count, when present, must be nonzero. Chapter 6 validates streams, packet timestamps, variable frame rate, rotation, duration, and decoder limits.

Media type is routing evidence, not proof

MediaType accepts a conservative lowercase type/subtype token without parameters, bounded to 127 bytes. image/png, audio/wav, and video/mp4 pass.

Version 1 requires:

Text  → text/*
Image → image/*
Audio → audio/*
Video → video/*

The IANA media type registry is the authoritative registry. RFC 6838 defines registration and top-level semantics.

This rule is deliberately narrower than the entire media ecosystem:

JSON/XML source may use application/*;
PDF and office documents can combine text/images;
multipart/container formats can contain several media kinds;
some vendor formats have unusual registered types;
content-type parameters such as charset exist.

Do not weaken version 1 to accept arbitrary strings. Add a versioned Document/Composite variant and tested content-type policy when the product needs those artifacts.

Filename extensions and Content-Type headers are attacker-controlled hints. Acquisition should:

enforce byte-size limits while streaming;
compute the digest;
inspect bounded magic/container structure;
compare declared/detected types under policy;
quarantine disagreements;
invoke a sandboxed/limited decoder;
attach verified decode facts as a derived result.

The envelope’s consistency check catches obvious internal mismatch; it does not identify the bytes.

Declared versus verified is a trust boundary

Every version-1 detail field starts with declared_. This naming prevents a dangerous shortcut:

upload header says 800 × 600
→ allocate/accept as 800 × 600
→ decoder expands a much larger or malformed image

Never allocate based only on declarations. A decoder runs under independent bounds and returns:

verified format
verified dimensions/duration/streams
decode implementation release
warnings/disagreements
derived output digest

Retain both declared and verified values. A disagreement is evidence for diagnostics/security, not a reason to overwrite history.

The same principle applies to user-provided timestamps, EXIF, WAV headers, MP4 atoms, HTTP headers, and language labels. Record source and verification stage.

A digest is identity/integrity evidence, not authenticity

Given the same bytes and SHA-256 algorithm, the content digest is stable. It supports:

content-addressed storage;
exact deduplication;
tamper detection on later read;
cache keys;
parent/derived links;
reproducible fixture identity.

It does not establish:

who created/captured the bytes;
whether the content is true;
whether metadata was forged;
whether a screenshot was edited before acquisition;
whether consent/license permits use;
whether a camera/device was trusted;
whether two visually/semantically equivalent encodings are the same artifact.

Chapter 2 adds acquisition receipts, trust assertions, signatures/content credentials where available, and provenance graphs. Even signed provenance proves statements about a chain; policy still decides what to trust.

Keep raw bytes out of manifest JSON

Inlining base64:

expands data by roughly one third before JSON overhead;
forces large allocations/copies;
makes line bounds/search/diffs poor;
couples manifest parsing to media transfer;
encourages logging sensitive bytes;
prevents independent content-addressed caching;
makes partial/range access harder.

Store encoded bytes in the artifact layer and place only digest/length/type in the envelope. Fetch with tenant authorization, maximum bytes, regular-file/object validation, and digest verification.

Never log the whole envelope blindly if source locators/labels later contain sensitive data. Define a safe summary separately.

Parse one bounded strict JSON record

The entrypoint is:

pub fn parse_json_line(input: &str) -> Result<Self, EvidenceError> {
    if input.len() > MAX_ENVELOPE_JSON_BYTES {
        return Err(EvidenceError::ManifestTooLarge { /* ... */ });
    }
    let envelope: Self = serde_json::from_str(input)
        .map_err(|error| EvidenceError::InvalidJson(error.to_string()))?;
    envelope.validate()?;
    Ok(envelope)
}

The 64 KiB bound is checked before deserialization. It bounds one record, not the whole file. A JSONL reader must also:

bound line buffering;
cap record count and total bytes;
reject/define blank-line policy;
report line number without echoing sensitive content;
stream instead of loading the full manifest;
decide whether one invalid line rejects the whole manifest.

The example filters blank lines for a trusted checked-in fixture. A public ingestion endpoint should make that policy explicit and reject ambiguous whitespace/BOM/encoding before line parsing.

Strict fields are compatibility protection

Every envelope/detail struct uses:

#[serde(deny_unknown_fields)]

This prevents a producer from sending:

{"trusted": true}

and an older reader silently ignoring it while downstream code assumes trust was honored. Strict parsing turns producer/consumer schema skew into a visible failure.

Forward compatibility does not mean “ignore everything new.” It means negotiate or migrate versions deliberately.

Validation order and typed failures

parse_json_line performs:

record byte bound;
strict JSON/Serde shape and validated newtype decoding;
exact schema-version check;
nonempty encoded artifact;
media-type/modality compatibility;
modality-specific declared-detail checks.

Errors distinguish:

manifest too large
invalid JSON/domain newtype
unsupported schema
empty artifact
media/modality mismatch
zero declared value
incomplete dimensions
invalid bounded label

Do not return a generic “bad media.” Typed failure categories feed metrics, quarantine decisions, client corrections, and security triage.

One nuance: invalid IDs/digests/media types are validated by custom Deserialize implementations, so they surface wrapped as InvalidJson at the JSON boundary. Direct constructors return their specific EvidenceError. This preserves one safe parse boundary while keeping domain constructors usable in Rust code.

Exhaustive processing keeps modality visible

Downstream code must match:

match &envelope.body {
    EvidenceBody::Text(details) => validate_text(details),
    EvidenceBody::Image(details) => schedule_image_decode(details),
    EvidenceBody::Audio(details) => schedule_audio_probe(details),
    EvidenceBody::Video(details) => schedule_video_probe(details),
}

When a future Document variant is added, exhaustive matches fail to compile until every consumer decides what it means. That is safer than a default branch that drops the artifact or treats it as text.

Common operations can use common fields:

authorize evidence ID
locate bytes by digest
enforce encoded size
record acquisition
emit safe inventory
attach lineage

Decode, normalization, locator, retrieval, evaluation, and generation remain modality-specific.

Schema evolution rules

Version the semantic record, not only Rust structs.

Additive change

An optional field can still be behaviorally breaking if consumers reject unknown fields. For the strict contract:

add it in a new version;
provide a version-1 → version-2 migration;
freeze migration output fixtures;
preserve original source bytes/record;
update all exhaustive consumers.

Renaming or changing units

Never reuse a field with new meaning. duration_ms cannot become microseconds. Create a new version and explicit conversion with overflow/rounding policy.

New modality

Add a new enum variant only with:

media-type policy;
declared/verified metadata distinction;
decoder resource/threat model;
locator coordinate system;
storage/retention policy;
capability and evaluation plan;
backward/forward migration behavior.

Canonical serialization

serde_json::to_string produces a stable result for this concrete struct in current tests, but this chapter does not claim a cryptographic canonical JSON standard. Hash the referenced bytes directly. If manifests themselves must be signed/hashed across implementations, select and test a defined canonicalization format in Chapter 2.

Security implications

Bound record, total manifest, count, ID/label, and encoded artifact sizes before expensive work.
Compute digests from streamed bytes; never trust a client digest.
Store/fetch under tenant authorization; a digest is not a secret bearer token.
Do not derive filesystem paths directly from IDs, filenames, URLs, or media types.
Reject unknown fields and schema versions.
Compare declared/detected media type; sandbox hostile parsers/decoders.
Cap decoded pixels, samples, duration, tracks, frames, nesting, metadata, and decompression ratio.
Treat declared dimensions/duration/charset as untrusted.
Prevent URL ingestion from reaching disallowed internal/link-local/metadata endpoints.
Preserve source/acquisition policy without exposing full locators in ordinary logs.
Quarantine malformed/mismatched artifacts rather than repeatedly retrying decode.
Scan/authorize derived artifacts independently; transformation does not make content trusted.
Do not expose raw embeddings/transcripts/OCR as if they were public when parents are restricted.
Apply retention/deletion transitively through derivation graphs.

Performance implications

The envelope is intentionally small and cheap. The expensive work is byte transfer, decode, resampling, frame extraction, OCR/ASR/embeddings, and model inference.

Design for:

streaming ingestion with incremental hash and byte cap;
content-addressed deduplication scoped by authorization/tenant;
bounded parallel decode queues by modality;
cache keys including input digest plus transform implementation/parameters;
no base64 copies in manifest paths;
lazy derived artifacts;
batch grouping by compatible shape/runtime without erasing evidence IDs;
cancellation cleanup;
separately measured encoded, decoded, and derived storage;
safe summaries rather than huge debug formatting.

Do not infer decode memory from size_bytes. A compressed image/video/audio artifact can expand dramatically.

Failure investigations

Failure: a PNG envelope parses but decode says JPEG

The manifest consistency check only compared image/png with Image; both share the image top-level type. Inspect magic/container probe and acquisition header. Quarantine the discrepancy, preserve both declarations, compute bytes’ digest, and do not route solely by suffix/type string.

Failure: image dimensions are swapped after decode

Declared encoded dimensions may precede orientation application. Preserve encoded dimensions, orientation metadata, and normalized dimensions separately. Regions must name their coordinate space/version.

Failure: audio duration differs from decoded sample count

Inspect container duration, codec delay/priming, truncation, sample rate, packet timestamps, and decoder warnings. Never overwrite the original declaration. Attach verified decode facts with implementation identity.

Failure: video has no audio but ingestion rejects it

Version 1 permits declared_audio_tracks: 0; zero is meaningful here. It rejects a declared video_tracks: 0 because the artifact was classified as video. If the container is a composite document/data artifact, use a future explicit variant rather than forcing it into video.

Failure: a new producer field disappears

With strict parsing it should fail, not disappear. Confirm every nested struct/enum uses deny_unknown_fields, version negotiation is active, and no intermediate generic JSON layer strips unknown fields before typed validation.

Failure: the same bytes appear under two evidence IDs

That is allowed. Inspect acquisition/source/tenant context. Content deduplication and logical evidence identity are different. Authorization must not leak one tenant’s existence through a global digest lookup.

Failure: repeated valid-looking digests point to missing bytes

The envelope is referential metadata, not transactional proof the artifact is present. Ingestion must commit bytes and manifest with an explicit consistency protocol, and verification must test missing/tampered object behavior.

What version 1 intentionally does not model

source URL/path/device identity or acquisition actor/receipt;
trust, consent, license, sensitivity, tenant, or retention;
derived artifacts and transformation graph;
locators, regions, time spans, text offsets, or source lines;
verified decoded format/properties;
language, encoding correctness, orientation, color, channel layout, codec, time base, or VFR;
multipart, PDF, office/document, 3D, haptics, sensor, or arbitrary application media;
content-type parameters;
perceptual hashes;
signatures/content credentials;
annotations, observations, claims, or citations.

These are not “extras” to store in a free-form map. They receive typed chapters/schemas next.

Exercises

Recall

What is the difference between encoded artifact and evidence envelope?
Why is OCR text not a replacement for an image?
Which fields are genuinely common to all four version-1 modalities?
Why are detail fields prefixed declared_?
What does a SHA-256 digest prove after verified read?
What does it not prove about truth/authorship?
Why must width and height be present together?
Why can a video legitimately declare zero audio tracks?
Why is acquisition time not video timeline time?
Why reject unknown future schema versions?

Implementation

Add a streaming JSONL reader with per-line, count, and total-byte limits.
Integrate mosaic-artifacts: write real fixture bytes, use returned digest/length/type, and verify bounded reads.
Add a Document variant for application/pdf with an explicit declared-detail policy.
Add a versioned migration from v1 to v2 without mutating source records.
Add property tests for valid evidence IDs and conservative media tokens.
Fuzz parse_json_line and assert it never panics or allocates beyond process policy.
Add a safe log summary that excludes future source locators.
Add a transactional ingestion test where manifest publication cannot precede artifact commit.
Add a quarantine record for declared/detected media disagreement.
Add per-tenant manifest lookup proving cross-tenant digest equality leaks nothing.

Design

Decide whether an animated GIF is image or video in your product; state locator/evaluation consequences.
Design a composite document envelope without flattening pages, text, and images.
Design acquisition-time uncertainty for a device with an unsynchronized clock.
Decide how deletion propagates from one parent to cached derivatives.
Design an authenticated manifest signature/canonicalization scheme.
Define which fields can appear in logs, metrics, traces, user exports, and restricted audit.

Solution guidance

Bytes are the preserved encoded object; the envelope is typed metadata/reference around it.
OCR loses pixels/layout/uncertainty and is a derived observation.
Version, logical ID, encoded digest/size/media type, source class, optional acquisition time, and tagged body.
They are unverified claims/hints used before bounded decode.
Recomputed equality strongly identifies byte equality/integrity under the algorithm assumptions.
Origin, truth, consent, license, and whether content was manipulated before hashing.
One dimension cannot define geometry and creates invalid states.
A silent video still has time-varying pictures; audio is optional.
They use different clock origins/uncertainty and serve different questions.
Ignoring new fields can silently discard security/semantic requirements.

For JSONL streaming, use a bounded buffered reader that aborts as soon as one line exceeds the cap; do not call an unbounded read_line then check. Track cumulative bytes/count and include line number in sanitized errors.

For PDF/composite media, resist HashMap<String, Value>. Define pages/resources plus later derived text/image children and locators. Make the security/decoder limits explicit before acceptance.

Check your understanding

Can two evidence IDs reference identical bytes?
Can one evidence ID change its digest in place?
Does media_type = image/png prove PNG decoding will succeed?
Why keep original bytes after normalization?
What does deny_unknown_fields protect?
Where should a content digest be computed?
Why is base64 in the manifest harmful?
Which type prevents text plus image dimensions?
Can declared_audio_tracks be zero?
When is a media-type parameter needed?
How will a claim eventually cite a specific region or time span?
Which missing fields are intentionally deferred to the provenance chapter?

Completion checklist

Authoritative references

The next chapter turns these four roots into a provenance graph. It adds verified transformations, parent/child lineage, precise locators, trust assertions, and reproducible derived-artifact identity—without changing what the original envelope claimed at acquisition.