Images — Decode, Orientation, Color, Regions, OCR, and Vision Embeddings

An image pipeline can return the right-looking thumbnail while producing the wrong evidence.

The file name can lie about the format. A tiny compressed file can expand into hundreds of millions of pixels. A phone photo can store pixels sideways and rely on EXIF orientation at display time. The same RGB triplet means different light under different color spaces. Transparent pixels can hide arbitrary RGB values. OCR can return plausible text from the wrong rectangle. A vision embedding can silently change when a resize filter, crop, color conversion, alpha background, model revision, or normalization constant changes.

This chapter builds a strict static-raster boundary in mosaic-evidence. It detects format from bytes, applies independent encoded/decode/working-set limits, reads orientation before decoding, records the color decision, produces oriented sRGB RGBA8 pixels, validates half-open pixel regions, attaches bounded OCR observations, creates explicit CHW model tensors, and binds every vision embedding to the model, preprocessing release, pixel digest, region, and dimension.

The implementation is intentionally honest about its current boundary. PNG, JPEG, and PNM are enabled; PNM exists for a dependency-free executable fixture. The path decodes one static image. It does not claim to validate every animated PNG frame, ICC workflow, HDR mastering intent, OCR engine, or learned vision model. Those capabilities require additional typed paths rather than quietly widening this one.

Learning objectives

By the end you will be able to:

distinguish encoded artifact identity from decoded pixel identity;
detect formats from content instead of trusting extensions or declared media types;
apply width, height, pixel, decoder-output, working-memory, and tensor limits independently;
explain why a decoder library allocation limit is not a complete memory sandbox;
separate stored dimensions from post-orientation dimensions;
implement all eight EXIF orientation transforms exactly once;
map half-open rectangles between encoded and oriented coordinate spaces;
distinguish color metadata, color conversion, and an explicit missing-metadata assumption;
explain why resizing and alpha composition have color-space semantics;
state the difference between straight and premultiplied alpha;
use a derived RGBA digest without confusing it with the uploaded-file digest;
keep OCR as a versioned, ranged observation rather than replacement image truth;
construct deterministic crop, resize, composite, scale, normalize, and CHW preprocessing;
bind an embedding to exact model and preprocessing contracts;
reject cross-model or cross-preprocessing similarity comparisons;
investigate decompression bombs, orientation errors, color shifts, bad OCR, and embedding drift;
identify what should move into a worker process before accepting untrusted production media.

Prerequisites and dependency order

Read these chapters first:

One evidence envelope without erasing modality stores the immutable encoded artifact digest and keeps declared dimensions explicitly unverified.
Provenance graphs, hashes, locators, trust, and transformations gives every derived image, OCR result, and embedding a parent locator and transformation receipt.
Text normalization, structure, language, and source coordinates defines how OCR text can enter the text pipeline without losing its source pixel region.
Quantization, memory, batching, and device placement provides the memory-accounting vocabulary used when tensors later enter an accelerator runtime.

Image decode must happen before OCR, region-level retrieval, vision embeddings, captioning, visual question answering, or image generation evaluation. Orientation and color must be decided before those derived observations are assigned coordinates or compared.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example image_pipeline

The example constructs a real binary PPM/PNM image in memory:

encoded: 29 bytes
stored dimensions: 3 × 2
stored pixels:
  red      green    blue
  yellow   magenta  cyan

It then:

detects PNM from the bytes;
decodes under tight format, geometry, memory, and tensor limits;
establishes sRGB and converts to oriented RGBA8;
hashes the exact oriented pixel buffer;
attaches a bounded OCR observation to [0,0,2,1);
crops the full image, resizes to 2 × 2, composites alpha over white, normalizes, and writes CHW values;
attaches a three-value fixture descriptor under the explicit model label fixture-channel-statistics-not-semantic@v1.

The descriptor exercises the embedding contract and L2 normalization. It is not a learned semantic vision embedding. Candle, mistral.rs, and ONNX Runtime chapters will supply real inference engines without changing this evidence contract.

The output includes:

{
  "decode_release": "mosaic-image-decode-v1+image-0.25+orientation+srgb+rgba8",
  "format": "pnm",
  "encoded_dimensions": [3, 2],
  "oriented_dimensions": [3, 2],
  "orientation": "no_transforms",
  "model_input": {
    "shape_chw": [3, 2, 2],
    "preprocessing_release": "mosaic-image-input-v1+triangle+srgb-encoded+straight-alpha-over"
  }
}

One image, several representations

Do not use one Image struct for every stage. The stages answer different questions:

Representation	Identity and coordinates	What it knows
Encoded artifact	SHA-256 of exact uploaded bytes; byte offsets	container/codec bytes and acquisition context
Decoder view	stored width/height and decoder color type	properties reported by the selected decoder
Oriented pixels	RGBA8 digest; top-left half-open pixel coordinates	applied orientation and declared color decision
Region	`(x, y, width, height)` in one named pixel space	crop/source location
OCR observation	region plus text, confidence, method, order, language	what one OCR release observed
Model input	CHW tensor plus preprocessing release and source region	exact numeric values sent to a model
Vision embedding	model/preprocess/pixel digest/region/dimension	normalized representation from one contract

An encoded PNG digest and an oriented RGBA digest are not competing hashes. They identify different artifacts. Two PNG files can encode identical pixels with different metadata or compression. Two decoder releases can conceivably produce different pixels from the same malformed or ambiguous file. A provenance graph should retain both identities and name the decoding transformation between them.

The chapter uses this derived-pixel release:

mosaic-image-decode-v1+image-0.25+orientation+srgb+rgba8

That human-readable value is only one component of executable identity. Reproducibility also needs the workspace lockfile, target/toolchain, feature set, policy, and source revision.

Step 1: make decode policy explicit

The boundary begins with a value, not hidden constants:

pub struct ImageDecodePolicy {
    pub max_encoded_bytes: usize,
    pub max_width: u32,
    pub max_height: u32,
    pub max_pixels: u64,
    pub max_decoder_bytes: u64,
    pub max_working_bytes: u64,
    pub max_tensor_pixels: u64,
    pub assume_srgb_when_unspecified: bool,
    pub formats: StaticFormatPolicy,
}

Each limit controls a different amplification path.

Encoded bytes

This is the request/object-store read bound. It prevents an unbounded input buffer, but does not constrain decoded pixels. A highly compressible image can be small on disk and enormous in memory.

Width and height

Per-axis limits prevent shapes that are unacceptable even when total pixels are moderate. An image that is one pixel high and tens of millions wide can stress resizers and indexing assumptions.

Pixel count

The checked product u64::from(width) * u64::from(height) constrains total spatial work. It is separate from dimensions so every multiplication is explicit and overflow-safe.

Decoder output bytes

ImageDecoder::total_bytes() estimates the decoded output required by the decoder. The chapter checks it against max_decoder_bytes before calling DynamicImage::from_decoder.

The Rust image::Limits API also receives strict maximum width and height plus max_alloc. Its documentation is important: width and height are strict, while max_alloc is best-effort and some decoders may not support every allocation limit. Treat it as defense-in-depth, not a process memory guarantee.

Postprocess working bytes

Orientation may require source and destination RGBA buffers at the same time. This implementation therefore checks:

RGBA bytes = width × height × 4
orientation working bound = RGBA bytes × 2

That is an exact bound for the two explicit vectors in this transformation, not a bound on every temporary inside the decoder or color-management implementation.

Tensor pixels

Model input has a separate bound because a valid source image should not authorize an arbitrarily large tensor. width × height for the requested model input must remain within max_tensor_pixels; the implementation then allocates exactly three f32 planes.

Absolute protocol ceilings

The library also rejects policies above hard protocol ceilings. A caller cannot accidentally configure “unlimited” by setting usize::MAX. Application defaults and tenant plans should normally be tighter.

For hostile production media, combine these checks with:

request and object-store bounds;
concurrency admission before decode;
a worker process/container with memory, CPU, wall-clock, and file-descriptor limits;
no network or unnecessary filesystem access in the decoder worker;
timeout-driven process termination;
decoder fuzzing and patched dependencies;
per-format telemetry and quarantine of repeated failures.

Rust memory safety does not make a decoder immune to denial of service, pathological CPU use, native library defects, or valid inputs that exceed your capacity plan.

Step 2: detect the format from bytes

The implementation creates an ImageReader around the byte slice and calls with_guessed_format(). The ImageReader documentation describes this as content-based format guessing. The detected ImageFormat is then mapped into the deliberately small RasterFormat enum and checked against the policy allowlist.

let mut reader = ImageReader::new(Cursor::new(bytes))
    .with_guessed_format()
    .map_err(...)?;
let image_format = reader.format().ok_or(RasterError::UnknownFormat)?;
let format = RasterFormat::from_image(image_format)
    .ok_or_else(...)?;
if !policy.formats.allows(format) {
    return Err(RasterError::FormatNotAllowed(...));
}

The file extension, HTTP Content-Type, evidence-envelope media type, and magic-byte detection are separate observations. A product should compare them and record disagreement. It should not select a decoder from an attacker-controlled extension and should not silently rewrite the acquisition record after sniffing.

An allowlist also constrains dependency surface. Adding GIF, WebP, TIFF, AVIF, HEIF, JPEG XL, SVG, PDF, or camera RAW is an engineering decision with new parsers, animation rules, color behavior, and security tests—not a checkbox added for convenience.

Static does not mean “first frame is enough”

This boundary is named StaticFormatPolicy, but PNG can carry animation through APNG extensions. The current path decodes the default/static image exposed by image; it does not inspect all frames and prove absence of animation. Therefore:

do not advertise this path as APNG-safe animation ingestion;
add a preflight/parser rule that explicitly rejects animated PNG when static-only evidence is required; or
route animation to a bounded frame/timeline decoder with duration, disposal, blending, and frame count semantics.

The same rule applies to every format that can contain multiple pages, images, layers, or frames. Silently selecting “page 1” destroys evidence.

Step 3: read geometry and orientation before transforming pixels

The decoder exposes stored dimensions and orientation. The ImageDecoder contract makes both available before full decode. The implementation validates stored dimensions, pixel count, and output bytes, then records the orientation.

There are eight orientation states:

State	Oriented dimensions
no transform	`W × H`
rotate 90°	`H × W`
rotate 180°	`W × H`
rotate 270°	`H × W`
horizontal flip	`W × H`
vertical flip	`W × H`
rotate 90°, then horizontal flip	`H × W`
rotate 270°, then horizontal flip	`H × W`

The Orientation documentation maps these states to EXIF orientation values. The two combined states are easy to implement incorrectly because transform order matters.

The companion does not call a “best effort auto-orient” helper and discard the history. It records the stored dimensions and orientation, constructs a new oriented buffer, and exposes the resulting dimensions. Tests exercise every orientation’s region mapping and an exact rotated pixel fixture.

Pixel mapping versus boundary mapping

A pixel coordinate addresses a discrete sample:

0 ≤ x < W
0 ≤ y < H

A half-open rectangle uses boundaries:

[x, x + width) × [y, y + height)

Boundary coordinates can equal W or H; pixel coordinates cannot. That is why the implementation has distinct map_pixel and map_boundary formulas. For a 90° clockwise rotation:

pixel:    (x, y) -> (H - 1 - y, x)
boundary: (x, y) -> (H - y, x)

To map a rectangle, transform all four boundaries, take minimum/maximum coordinates, construct the new half-open rectangle, and validate it in the oriented dimensions. Applying the pixel formula to rectangle edges creates classic one-pixel shifts.

Apply orientation exactly once

After physical orientation, every OCR region, crop, tensor, embedding locator, and UI overlay in this chapter uses the oriented top-left coordinate space. If the derived pixels are re-encoded, clear or rewrite orientation metadata so a viewer does not rotate them a second time.

A production schema should name the space in the type itself—for example OrientedRgbaPixelRegion—when encoded and oriented regions can coexist. The compact chapter type relies on method boundaries and documentation; the provenance locator must record the same coordinate-space contract.

Step 4: make the color decision observable

RGB values are not colors without primaries, transfer characteristics, matrix/range semantics, and sometimes an ICC transformation context. The W3C PNG Third Edition specification defines how PNG color chunks, including cICP, iCCP, sRGB, and gamma/chromaticity information, participate in color interpretation. It also specifies alpha as full-range opacity associated with color samples.

The image crate represents coding-independent code points through Cicp. The boundary records the source value and chooses one of three outcomes:

pub enum ColorDecision {
    AlreadySrgb,
    ConvertedToSrgb,
    AssumedSrgbBecauseUnspecified,
}

If the source is already Cicp::SRGB, retain the samples.
If primaries or transfer are unspecified, reject unless policy explicitly permits an sRGB assumption.
If the metadata describes another supported space, convert to sRGB RGBA8.

An assumption is not a conversion. set_color_space(Cicp::SRGB) labels previously unspecified samples under a policy decision; it cannot recover missing author intent. Recording AssumedSrgbBecauseUnspecified lets evals slice those cases and lets a later ICC-capable pipeline replace the decision.

What this implementation does not promise

The color path is deliberately practical, not a reference color laboratory:

the current target is SDR sRGB RGBA8;
conversion support is limited by image 0.25 and the enabled codecs;
it does not preserve HDR precision or mastering metadata;
it does not expose rendering intent, black-point compensation, or every ICC profile detail;
malformed/conflicting metadata needs frozen fixtures and an explicit precedence policy;
source_color_space is diagnostic text, not a durable interchange schema.

Products where color is evidence—medical imaging, scientific measurement, print proofing, remote sensing—need domain formats, calibrated displays, higher precision, validated color management, and domain-specific error budgets. Do not reuse this sRGB convenience boundary as a measurement system.

Resizing is a color operation

The companion’s model preprocessing uses a triangle filter directly over encoded sRGB sample values. That exact behavior is included in:

mosaic-image-input-v1+triangle+srgb-encoded+straight-alpha-over

Interpolating encoded sRGB is deterministic and common, but it is not the same as converting to linear light, filtering, and re-encoding. Edges and brightness can differ. If a target model’s processor uses linear-light resize—or a specific library’s bicubic coefficients—implement and name that exact contract. “224 square” is nowhere near a complete preprocessing specification.

Step 5: define alpha composition

RGBA8 carries three color samples and one alpha sample. Two representations are frequently confused:

straight alpha stores unassociated RGB plus alpha;
premultiplied alpha stores RGB already multiplied by alpha.

The DynamicImage::into_rgba8() path gives the companion straight RGBA samples. A three-channel model cannot receive transparency directly, so preprocessing must choose a background. The example chooses white and computes each channel with integer round-to-nearest:

composited = (foreground × alpha + background × (255 - alpha) + 127) / 255

The background is part of ModelImageInput. Black, white, checkerboard, learned alpha features, and “discard alpha” produce different tensors. Transparent pixels can contain nonzero hidden RGB; using RGB without alpha can expose those values to the model.

This implementation composites in encoded sRGB sample space because that is the named version-1 contract. Physically correct compositing normally occurs in linear-light space. A future linear-srgb release should convert, composite, then encode/normalize, with golden fixtures proving the change.

Also decide whether resize occurs before or after composition. This chapter resizes straight RGBA, then composites. Other processors composite first. The order affects partially transparent edges and must be frozen with the model release.

Step 6: hash the derived pixels

After color decision, RGBA8 conversion, and orientation, the implementation hashes the raw row-major buffer:

SHA-256(R0,G0,B0,A0,R1,G1,B1,A1,...)

Width, height, pixel format, coordinate origin, row order, color contract, decoder release, and orientation are not serialized into that byte buffer. Therefore the digest is safe only inside the named derived-artifact schema. A production content-addressed record should hash or bind a domain-separated manifest such as:

schema = oriented-rgba8-v1
width
height
row_order = top-to-bottom
channel_order = RGBA
color = sRGB
alpha = straight
pixel_bytes_sha256
decode_release

Do not replace the source artifact digest with this value. Preserve:

exact encoded bytes/digest for acquisition, signatures, and later re-decode;
derived pixel artifact/digest for OCR, crops, embeddings, cache keys, and model inputs;
a provenance edge containing the decoder release and policy/parameter digest.

Hash equality proves equality of the bytes under the schema. It does not prove camera authenticity, capture time, authorship, consent, license, semantic equivalence, or absence of manipulation.

Step 7: use validated half-open regions

PixelRegion is:

pub struct PixelRegion {
    pub x: u32,
    pub y: u32,
    pub width: u32,
    pub height: u32,
}

Validation requires:

nonzero width and height;
checked x + width and y + height;
end boundaries no greater than the verified oriented dimensions.

Half-open coordinates compose cleanly:

width  = end_x - start_x
height = end_y - start_y

Adjacent boxes can share a boundary without sharing a pixel. A full image is exactly (0, 0, width, height).

Rectangles are sufficient for crops and many OCR boxes, but not every task. Text can be rotated; segmentation masks are irregular; object detectors may produce normalized floating-point boxes; PDF/image hybrids may have physical units. Add typed polygons, masks, transforms, and unit conversions rather than overloading PixelRegion.

Never validate a locator against the width/height declared by the untrusted evidence envelope. Use the dimensions established by the decoder. Preserve the declaration separately so disagreement can be investigated.

Step 8: attach OCR as observation, not truth

The boundary stores:

pub struct OcrObservation {
    pub text: String,
    pub region: PixelRegion,
    pub confidence_basis_points: u16,
    pub method_release: String,
    pub reading_order: u32,
    pub language: Option<LanguageLabel>,
}

The implementation rejects:

empty/whitespace-only text;
NUL;
text above the byte bound;
out-of-range rectangles;
confidence above 10,000 basis points;
malformed method releases;
duplicate reading-order positions.

Then it sorts observations by reading order. That order is a model output, not a geometric theorem. Multi-column documents, tables, vertical scripts, captions, and diagrams need structured layout graphs. Sorting only by y then x is not a universal reading-order algorithm.

Confidence also needs a contract. The field is an integer transport format. It does not become a calibrated probability because an engine returned 0.96. Evaluate confidence against character/word error and task failures, sliced by language, font, resolution, rotation, blur, contrast, layout, and source type. Keep engine/version identity exact.

OCR output should flow through the text boundary as a derived text artifact:

oriented RGBA pixels + region
  └─ OCR engine/model/parameters
      ├─ raw engine output artifact
      ├─ bounded text observation
      └─ provenance edge back to pixel region
          └─ strict text normalization/chunking/tokenization

Do not overwrite pixels with OCR text. Do not cite only the normalized string. A rendered citation must be able to reopen the source image and draw the oriented pixel box that supported the claim.

For serious OCR, extend the schema with:

word/line/block hierarchy;
quadrilaterals or polygons;
per-token alternatives and confidence;
script and orientation;
table cells and reading graph;
original engine response digest;
preprocessing operations;
human correction as a new asserted derivative, never an in-place mutation.

Step 9: construct the exact model tensor

The model input contract contains:

pub struct ModelImageInput {
    pub width: u32,
    pub height: u32,
    pub mean: [f32; 3],
    pub standard_deviation: [f32; 3],
    pub alpha_background_srgb: [u8; 3],
}

The implementation performs these operations in order:

validate the source region in oriented coordinates;
validate nonzero target dimensions and tensor pixel budget;
reject non-finite means and non-positive/non-finite standard deviations;
copy the RGBA buffer into a typed image;
crop the selected region;
resize to the exact target with triangle filtering;
composite each straight-alpha pixel over the explicit background;
convert byte samples to [0,1];
normalize (value - mean) / standard_deviation;
write channel-first [3, height, width] values;
reject any non-finite output.

The resulting ImageTensor preserves:

shape;
values;
oriented source region;
preprocessing release.

Real processors may additionally require aspect-ratio preservation, letterboxing, center/random crops, multiple resolutions, patch alignment, BGR order, integer tensors, different ranges, or model-specific interpolation. Implement the model’s checked processor contract; do not infer it from training folklore.

The current method clones the complete RGBA buffer before cropping. That makes ownership and bounds clear for the first implementation, but is not the final throughput design. A production path can use a borrowed view, fused crop/resize/normalize, reusable buffers, or accelerator preprocessing after benchmarks prove equivalence. Optimization must retain golden tensor tests.

Step 10: bind vision embeddings to their complete contract

A vector without provenance is not a reusable embedding. The companion stores:

pub struct VisionEmbedding {
    pub model_release: String,
    pub preprocessing_release: String,
    pub source_pixel_sha256: Sha256Digest,
    pub source_region: PixelRegion,
    pub values: Vec<f32>,
}

Construction validates:

nonempty expected dimension below the protocol maximum;
exact vector length;
finite elements;
nonzero finite L2 norm;
valid model/preprocessing labels;
a region inside the verified image.

It L2-normalizes once. Cosine similarity then becomes the dot product, clamped to [-1,1] for floating-point tolerance. Comparison rejects different model releases, preprocessing releases, or dimensions.

This follows the contract implied by contrastive systems such as CLIP: image and text representations are meaningful because a specific model learned a shared space under a specific training/objective setup. The paper does not make vectors from arbitrary models mutually comparable. A dimension match is not semantic compatibility.

In production, also bind:

model artifact manifest/revision and runtime release;
processor configuration digest, tokenizer for text queries, and prompt/template when applicable;
numeric dtype and normalization behavior;
whole-image versus crop aggregation;
tenant/access-policy scope;
created time and invalidation status;
embedding schema version.

Changing preprocessing can require a complete re-embedding even if the model weights did not change. Use separate index namespaces by contract. During migration, dual-write or rebuild, evaluate old and new retrieval on the same frozen set, then switch deliberately.

Embedding quality is an eval question

Cosine similarity is a numeric operation, not proof that retrieval is useful. Evaluate:

Recall@k and nDCG on labeled image/text and image/image queries;
exact model/preprocessing version slices;
crop versus whole-image cases;
OCR-heavy screenshots versus natural images;
near duplicates and visually similar but semantically distinct negatives;
language, accessibility, and domain slices;
abstention/no-match thresholds;
retrieval latency, index memory, and re-embedding cost.

A harness can combine visual embedding retrieval with OCR lexical retrieval, metadata filters, and a reranker. Later chapters will make fusion, diversity, and citation policies explicit.

Failure investigations

“A 200 KiB upload exhausted memory”

Inspect stored dimensions, pixel count, decoder total_bytes, concurrency, and worker RSS. Verify the request was admitted before allocation and all arithmetic was checked. Do not fix it by raising only the encoded-byte limit. Add a decompression-bomb fixture and a process memory limit.

“OCR boxes are shifted or rotated”

Check whether OCR ran before orientation while the UI displays after orientation. Inspect the stored orientation, stored/oriented dimensions, coordinate-space label, and whether metadata was applied twice. Use a nonsymmetric numbered pixel fixture; square photos hide rotation bugs.

“The image looks washed out”

Inspect source color metadata, ColorDecision, conversion support, precision loss, alpha handling, and viewer behavior. Compare known color patches. Do not patch the symptom with arbitrary contrast.

“Transparent logos have dark halos”

Check straight versus premultiplied alpha, resize/composite order, background, and encoded-space filtering. Hidden RGB around zero alpha often reveals the mismatch. Freeze edge fixtures.

“Retrieval changed after a harmless refactor”

Compare model artifact identity, preprocessing release, decoder version/features, orientation, color decision, resize filter, crop, mean/std, alpha background, tensor order/dtype, and embedding normalization. A source-file hash alone cannot explain the drift.

“Two vectors have 768 elements, so why reject them?”

Dimension is only shape. If model or preprocessing identity differs, the axes do not have a shared learned meaning. Store them in different index namespaces or use a validated cross-space mapping.

“The PNG passed static ingestion but contains motion”

The current boundary has exposed its documented limitation. Add explicit APNG detection/rejection or route the file to the video/animation timeline. Do not silently accept the default frame as the whole evidence.

Testing strategy

The raster module has focused tests for:

actual PNM byte detection and decode;
encoded-size and unknown-format rejection;
detected-but-disallowed formats;
independent width, pixel, and working-memory failures;
all eight orientation rectangle mappings;
exact rotation of a nonsymmetric pixel grid;
zero, overflowing, and out-of-bounds regions;
OCR ordering and malformed observations;
deterministic CHW preprocessing and alpha background;
invalid tensor dimensions/statistics/budgets;
embedding normalization and contract mismatch;
empty, non-finite, zero-norm, and dimension-mismatched embeddings.

The next production test corpus should add:

JPEG fixtures carrying all eight EXIF orientation values;
PNG/JPEG color fixtures with sRGB, cICP, ICC, gamma/chromaticity, missing, and conflicting metadata;
transparent-edge fixtures with hidden RGB;
corrupt/truncated files and declared-size bombs;
APNG detection cases;
decoder differential tests across pinned upgrades;
golden tensor digests from real model processors;
OCR fixtures across layouts/languages/degradation;
learned embedding fixtures and retrieval evals.

Property tests should generate valid dimensions/regions and assert that mapping remains inside the oriented space, preserves rectangle area for these orthogonal transforms, and round-trips through the inverse orientation. Fuzz the byte boundary in an isolated fuzz target.

Exercises

Exercise 1: add a typed pixel-artifact manifest

Create OrientedRgbaArtifactV1 with schema, dimensions, row/channel order, color/alpha semantics, pixel digest, decoder release, and source evidence ID. Compute a domain-separated manifest digest. Prove that changing dimensions or semantics changes identity even when pixel bytes are the same.

Exercise 2: invert orientation mapping

Implement inverse() for every RasterOrientation. Generate valid rectangles for varied nonsquare dimensions; map forward then backward and assert equality. Include one-pixel edge boxes.

Exercise 3: reject animated PNG explicitly

Add a bounded preflight that recognizes APNG animation control before static decode. Freeze static PNG, animated PNG, malformed chunk, oversized chunk, and truncation fixtures. Record the rejection as a typed outcome.

Exercise 4: linear-light preprocessing

Add a new preprocessing release that decodes sRGB transfer, composites and resizes in linear light, then applies the target model’s numeric normalization. Compare edge/gradient tensors with version 1. Do not change version-1 outputs.

Exercise 5: OCR calibration

Create a dataset with ground-truth text, boxes, language/layout slices, and engine confidences. Measure character error rate, word error rate, box IoU, and calibration. Choose an escalation threshold from measured cost, not intuition.

Exercise 6: real vision embeddings

Use the later Candle or ONNX path to load a pinned CLIP-like image encoder and processor. Embed frozen whole images and crops, store the full contract, and evaluate Recall@k against the fixture descriptor baseline.

Solution sketches

Exercise 1

Use a binary, length-prefixed, domain-separated hash—not incidental JSON serialization. Validate dimensions and digest before hashing. Include a literal prefix such as mosaic-oriented-rgba-artifact-v1\0; append every field in a fixed order and fixed integer endianness. Test one-field mutations.

Exercise 2

The inverse of 90° is 270°; 180° is itself; horizontal/vertical flips are themselves. Combined orientation inverses deserve table-driven tests rather than intuition. Remember that width/height swap between forward and inverse calls.

Exercise 3

PNG chunks are length, type, data, and CRC. Parse with checked offsets and a strict byte/chunk bound; do not scan for the ASCII text acTL anywhere in compressed data. A valid animation control chunk routes away from the static path. A malformed container rejects.

Exercise 4

Name the release and freeze conversion coefficients/implementation. Convert straight color samples to linear values, handle alpha consistently, filter, composite according to the selected order, and only then produce the model’s channel values. Compare numerical tolerances explicitly.

Exercise 5

Confidence bins need enough samples and fixed definitions. Preserve engine version and slice keys. Calibration can be useful even when raw recognition quality is unchanged because it improves routing. Human review becomes a new observation linked to the original region.

Exercise 6

Pin weights, processor, runtime, dtype, device behavior, and output normalization. Compare only vectors from the same contract. A real learned model should beat a three-channel statistic on semantic retrieval, but the result is not accepted until the frozen eval demonstrates it.

Completion checklist

References

Rust image crate: ImageReader, Limits, ImageDecoder, DynamicImage, and Orientation
PNG Specification, Third Edition — container, color, alpha, and animation requirements
CIPA DC-008-Translation-2019, Exchangeable image file format for digital still cameras: Exif Version 2.32
Learning Transferable Visual Models From Natural Language Supervision (CLIP)
W3C Web Annotation Data Model — selectors and annotations for resources
C2PA Technical Specification — signed content provenance and ingredients; not a substitute for application policy
Companion implementation: crates/mosaic-evidence/src/raster.rs and crates/mosaic-evidence/examples/image_pipeline.rs

What comes next

Images have two spatial axes and sometimes orientation metadata. Audio introduces a sample clock, channels, codec delay, resampling, time ranges, speech, and acoustic events. The next chapter builds that boundary without pretending container duration, sample indexes, transcript timestamps, and wall-clock time are one coordinate system.