Day 14 — Decode and Sample Video

Today Cutroom turns a video container into a plan. Planning first prevents a ten-minute file from silently becoming thousands of decoded frames and an unbounded model bill.

Run the pipeline model

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example video_pipeline

Inspect duration, timescale, track metadata, sample count, and transformation identities.

Separate container time from wall time

Video timestamps live in track-specific timebases. Represent raw timestamps with their timescale and convert using checked integer arithmetic. Variable-frame-rate video makes frame_number / fps an unsafe locator.

pub struct MediaTime {
    pub ticks: i64,
    pub timescale: u32,
}

Validate nonzero timescale, nonnegative duration where required, monotonic decode order, and bounded conversion. Keep presentation time distinct from decode order.

Build a sampling policy

Start with three signals:

• regular interval samples for coverage;
• scene-cut samples for change;
• user-supplied regions or timestamps for intent.

The policy returns a SamplePlan before decoding pixels. Deduplicate nearby requests and enforce maximum frames, maximum decoded bytes, and deadline.

Break it deliberately

• a zero timescale;
• variable frame rate with a constant-fps assumption;
• an hour-long video under a one-frame-per-second policy;
• rotation metadata ignored during frame extraction;
• a sample timestamp after the media duration.

Predict whether each must be rejected, clamped, or recorded as degraded evidence.

Completion proof

Given duration and policy, calculate the maximum samples and approximate decoded memory before running. Then compare with the emitted plan. A plan that cannot state its upper bound is not ready to execute.

Deep reference: Video demuxing, clocks, cuts, motion, and sampling.

Next: Day 15 — Specialist Signals →.