Day 11 — Screenshots and Voice Notes

Today Signal Room learns from a dashboard screenshot and an on-call voice note. Neither is sent to a general model as an opaque blob first. You create smaller observations that can be inspected and tested.

Run both pipelines

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example image_pipeline
cargo run -q -p mosaic-evidence --example audio_pipeline

Image pipeline

The steps are explicit:

encoded bytes → decode → orient → color-normalize → regions → OCR/vision observations

Store the transformation record. Pixel coordinates from the original screenshot are meaningless after crop and resize unless you can map them back.

An OCR result is an observation:

pub struct TextRegion {
    pub text: String,
    pub bounds: Rect,
    pub confidence: f32,
    pub producer: ModelRelease,
}

It is not automatically a trusted fact. Low confidence, tiny text, rotated labels, and model release changes belong in tests.

Audio pipeline

container → decode → channel policy → resample → timestamped speech/events

Use integer sample positions or explicit time units rather than accumulating floating-point seconds. Retain the original clock and the resampling map so a quoted transcript span can return to the source.

Run experiments with stereo-to-mono policy, 44.1 kHz to 16 kHz resampling, clipped audio, and silence. Record whether the change affected acquisition, speech recognition, or later reasoning.

Join only at the evidence layer

The screenshot and voice note become two envelopes with different locators. Signal Room may correlate them later, but it must never pretend a voice timestamp is a pixel coordinate or that OCR confidence is transcription confidence.

Completion proof

Add one adversarial image and one adversarial audio fixture. Your report must retain source digest, transform chain, observation producer, confidence, and reversible locator.

Deep references: Images and Audio.

Next: Day 12 — Align the Timeline →.