Video — Demuxing, Clocks, Variable Frame Rate, Cuts, Motion, and Sampling

Video is not an ordered folder of pictures.

It is a timed program. A container multiplexes tracks. A video track carries codec access units in decode order. Some frames depend on frames presented later, so decode order and presentation order can differ. Every sample has a duration under a rational media timescale. The duration can vary. Keyframes describe decoder entry points, not automatically the most informative visual evidence. Rotation, aperture, aspect ratio, color metadata, edit lists, and discontinuities can change what a viewer sees without changing the encoded sample bytes.

The common shortcut—timestamp = frame_number / fps—fails as soon as a recording is variable-frame-rate, contains reordered pictures, begins after an edit, drops frames, or crosses a timeline discontinuity. The shortcut can cite the wrong visual moment while returning a plausible thumbnail.

This chapter implements a narrower and checkable boundary:

generate and parse a real non-fragmented ISO BMFF/MP4 container;
choose a video track explicitly;
preflight sample-table sizes before payload allocation;
retain decode timestamp, presentation timestamp, duration, sync status, size, and digest for every opaque encoded sample;
validate non-overlapping half-open presentation intervals;
analyze bounded decoded luma frames with a 16-bin cut signal and deterministic block-SAD motion search;
sample by presentation-time coverage while retaining why each frame was selected;
report timeline gaps instead of assigning them to a nearby frame.

The implementation does not claim that arbitrary H.264, H.265, or VP9 bytes decode. The MP4 fixture contains genuine boxes and sample tables but intentionally opaque toy sample payloads. A production decoder is a separate native, process, service, or browser boundary. That separation is important: successfully demuxing a payload is not evidence that the payload is decodable, safe, color-correct, or visually meaningful.

Learning objectives

By the end you will be able to:

distinguish container, track, encoded sample, codec access unit, decoded frame, and rendered frame;
explain the different jobs of decode timestamp (DTS), presentation timestamp (PTS), duration, timescale, composition offset, and edit list;
explain why a frame rate is a summary, not a coordinate system;
define a frame’s presentation interval as [PTS, PTS + duration);
preserve exact integer ticks and derive outward-rounded microsecond coverage without drift;
identify the roles of MP4 stts, ctts, stss, stsz, and stco/co64 sample tables;
choose among multiple video tracks rather than relying on map iteration or “first track”;
preflight track count, sample count, sample size, total sample bytes, timescale, and dimensions;
keep payload identity and track-manifest identity separate from the container digest;
explain why sync samples are useful decoder entry points but weak semantic summaries;
define decoded-frame identity over pixels, geometry, timing, and sample association;
detect large distribution changes without calling every motion event a cut;
recover bounded translational motion with deterministic block matching;
recognize when camera motion, fades, flashes, noise, and low-texture regions fool simple detectors;
build a sampling plan that combines interval coverage, cuts, sync samples, first, and last frames;
preserve selection reasons and uncovered target times;
isolate untrusted parsers and decoders behind resource and process boundaries;
design the next adapter for fragmented MP4, FFmpeg, GStreamer, WebCodecs, or platform codecs.

Prerequisites and dependency order

Read the earlier Part 4 chapters in order:

Evidence envelopes define immutable encoded identity.
Provenance graphs distinguish acquisition from derived observations.
Text normalization shows why derived views need exact mappings back to sources.
Image decoding establishes bounded pixels, color decisions, and pixel-space observations.
Audio clocks uses integer sample coordinates and outward-rounded time coverage.

Also review blocking/media scheduling, content-addressed storage, and testing under faults. Demuxing and decoding are bounded CPU, memory, and I/O jobs, not async event-loop work.

Video normalization precedes audiovisual alignment, shot segmentation, object tracking, video embeddings, multimodal retrieval, captioning, edit planning, generation review, and Cutroom.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example video_pipeline

The example creates a 752-byte MP4 with one 1,000-tick-per-second H.264-labelled track. The track has four samples:

Sample	DTS	Composition offset	PTS	Duration	Sync
1	0	1,000	1,000	1,000	yes
2	1,000	2,000	3,000	1,000	no
3	2,000	0	2,000	1,000	no
4	3,000	1,000	4,000	2,000	yes

Decode order is 1, 2, 3, 4. Presentation order is 1, 3, 2, 4. The final two-second sample makes the duration distribution variable. Presentation coverage is [1,000, 6,000) ticks with no gaps.

The example then supplies four small decoded-luma fixtures in presentation order. The first transition is a one-pixel horizontal translation. The bounded block search recovers +1,000 milli-pixels in X with zero residual. The next transition changes the luma distribution enough to produce a cut score of 9,688 basis points. Sampling at one-second intervals selects all four frames and records merged reasons:

sample 1: first + interval_coverage + sync_sample
sample 3: interval_coverage
sample 2: interval_coverage + cut_boundary
sample 4: last + interval_coverage + sync_sample

Those reasons are part of the artifact. “The system chose sample 2” is less useful than “sample 2 covers the three-second target and is the first frame after a detected cut.”

First principles: identify every video layer

Encoded artifact

These are the exact acquired bytes. Their SHA-256 digest identifies the upload, fetch, or capture. The filename and declared media type remain declarations.

Container

ISO BMFF/MP4, Matroska/WebM, MPEG transport stream, QuickTime, AVI, and other containers describe tracks, timing, indexes, sample locations, metadata, and sometimes edits. A container parser demultiplexes; it does not necessarily decode.

Track

A container may contain a main camera, alternate angle, thumbnail track, depth map, alpha track, audio tracks, captions, and metadata. “Pick the first video track” is unstable if the parser stores tracks in a hash map, and semantically wrong when multiple angles exist. The companion requires explicit track selection when more than one video track is present.

Encoded sample or access unit

An MP4 sample is a byte range plus timing and dependency metadata. For common video tracks it often contains one codec access unit, but container sample and decoded display frame are not universal synonyms. A sample can fail to decode even if all of its container fields parse.

Decoded frame

A decoder turns codec data into pixel planes and metadata. YUV subsampling, coded dimensions, visible aperture, bit depth, color primaries, transfer function, matrix coefficients, range, rotation, and pixel aspect ratio all matter. A decoder may reorder output relative to input.

Rendered frame

Rendering may crop, rotate, scale, convert color, composite alpha, apply display transforms, and tone-map HDR. Evidence derived from decoded luma is not automatically evidence about the final displayed RGB appearance.

WebCodecs makes this distinction concrete: a VideoFrame has coded size, visible rectangle, display size, rotation, flip, color space, timestamp, and duration. Its current specification defines timestamp and duration in microseconds and requires resource release because decoded media can exhaust system resources. See the WebCodecs VideoFrame model.

The three clocks people accidentally merge

Decode time

DTS says when an encoded frame must enter the decoder. Predictive codecs can require a future display frame before decoding an intervening B-frame. Decode order must therefore remain monotonic even when presentation order is not.

Presentation time

PTS says when a decoded frame should be rendered. Presentation order is the ordering by PTS. The W3C Media Source Extensions specification defines a coded frame as having PTS, DTS, and duration, and defines a presentation interval as the half-open interval [PTS, PTS + duration). See MSE definitions.

Wall time

Wall-clock capture time is a separate coordinate: UTC, a monotonic device clock, RTP/NTP mapping, or recording-session time. Container media time may start at zero or another tick while wall time is hours or years later. Joining surveillance clips, live streams, or meeting recordings requires an explicit mapping with uncertainty. Do not store wall time in a PTS field.

Timescales are rational, not floating-point FPS

A track timescale T means T ticks per second. A sample with presentation tick p and duration d covers:

[p / T, (p + d) / T) seconds

Keep p, d, and T as integers. For a human-facing microsecond view, round the start down and the exclusive end up:

start_us = floor(p × 1,000,000 / T)
end_us   = ceil((p + d) × 1,000,000 / T)

That interval covers the original rational interval. It is not an invertible replacement for the ticks. At T = 3, one tick is 333,333⅓ microseconds. Repeatedly adding a rounded 333,333 loses a microsecond every three frames. The companion uses i128, Euclidean division, checked arithmetic, and signed ticks so a pre-edit or composition timeline can remain below zero.

Why average FPS lies

sample_count / duration can report 30 FPS for a track containing short bursts, long still holds, and missing periods. It describes aggregate density, not individual frame boundaries. Two files with the same average FPS can present different frames at 2.5 seconds.

Variable-frame-rate should be determined from sample durations and timeline structure, not a container’s nominal rate field. Apple’s QuickTime documentation has an explicit video variable-frame-rate indication, but downstream systems still need the actual per-sample timing.

MP4 sample tables: the minimum useful mental model

For a classic non-fragmented MP4 track, these tables answer different questions:

Box/table	Question
`stts`	How many decode-order samples use each duration?
`ctts`	What composition offset maps each decode time to presentation time?
`stss`	Which samples are sync samples/random-access candidates?
`stsz`	What is each sample’s encoded byte size?
`stsc`	How are samples grouped into chunks?
`stco` / `co64`	Where do chunks begin in the file?

The companion uses mp4 0.14 to parse those structures. Mp4Sample exposes start_time, duration, rendering_offset, is_sync, and bytes. The wrapper names start_time as DTS and computes:

PTS = DTS + rendering_offset

The wrapper does not flatten the result into a frame index.

Edit lists and authored movie time

The current boundary records the track’s raw media timeline. It does not claim to apply every ISO BMFF edit-list behavior, empty edit, dwell edit, movie-timescale conversion, or gapless splice. A player can therefore present an authored movie timeline different from this raw track timeline.

If exact player-equivalent timing is required, add an edit-list normalization stage with:

the selected movie and media timescales;
every edit segment and media-time value;
handling for empty edits and negative media time;
an exact piecewise mapping from raw PTS intervals to movie intervals;
fixtures compared against an independent player or probe;
a new transformation release and timeline digest.

Silently calling raw track PTS “video time” would hide this limitation.

Fragmented MP4

Fragmented MP4 stores runs in moof/traf/trun structures and can stream fragments over time. The chapter’s deterministic boundary rejects fragmented files. Adding them requires sequence, default-sample, base-offset, discontinuity, and partial-input rules. Rejection is safer than accidentally treating one fragment as the whole program.

Step 1: define independent demux limits

VideoDemuxPolicy contains separate ceilings for:

encoded file bytes;
total track count;
selected-track sample count;
bytes in one sample;
bytes across all selected samples;
timescale;
width and height.

These are not interchangeable. A tiny file can declare pathological counts. A normal sample count can contain one huge access unit. Small dimensions do not bound encoded metadata. A high timescale can overflow naive timestamp multiplication.

Protocol ceilings cap even caller-provided policy values. This prevents a configuration mistake from turning usize::MAX into a promise the process cannot keep.

let policy = VideoDemuxPolicy {
    max_encoded_bytes: 256 * 1024 * 1024,
    max_tracks: 16,
    max_samples: 2_000_000,
    max_sample_bytes: 64 * 1024 * 1024,
    max_total_sample_bytes: 2 * 1024 * 1024 * 1024,
    max_timescale: 100_000_000,
    max_dimension: 16_384,
};

Production values should come from a tenant/product policy and admission budget. The defaults are examples, not universal safe values.

Step 2: sniff, parse, and select deliberately

The wrapper requires an ftyp box at the strict boundary, parses from a bounded byte slice, rejects fragmented MP4, bounds the number of tracks, and enumerates video track IDs. If there is one video track, omission is unambiguous. If there are several, omission is an error:

let track = demux_mp4_video(&encoded, Some(2), policy)?;

This API prevents an implementation from changing track choice when hash seeds, insertion order, metadata order, or a new alternate track changes.

The current codec metadata boundary recognizes H.264, H.265, and VP9 sample entries supported by the parser. Recognition means “the sample entry says this codec,” not “the payload is valid.”

Step 3: preflight sizes before payload reads

The wrapper inspects stsz before calling read_sample.

For a fixed-size table:

total = fixed_sample_size × sample_count

For a variable-size table, it requires one size entry per declared sample, checks each against the per-sample bound, and sums with checked arithmetic. Only then does it permit the parser to allocate and read a sample payload. It checks actual returned sizes again.

This ordering matters. Validating bytes.len() after an unbounded parser allocation is not a memory limit. Preflight still does not make an in-process parser invulnerable: hostile nested box counts, parser defects, or decompression in later stages require process isolation and OS limits.

Step 4: retain both orders and bind identity

Each DemuxedVideoFrame stores:

pub struct DemuxedVideoFrame {
    pub sample_id: u32,
    pub decode_timestamp_ticks: u64,
    pub presentation_timestamp_ticks: i64,
    pub duration_ticks: u32,
    pub is_sync: bool,
    pub payload_size_bytes: u32,
    pub payload_sha256: Sha256Digest,
}

The track preserves frames in decode order. A separate method returns references sorted by (PTS, sample_id). It validates that every duration is positive and that presentation intervals do not overlap. Gaps remain legal and observable.

There are three related hashes:

the container digest identifies all encoded input bytes;
each payload digest identifies one sample’s bytes;
the track-manifest digest binds container digest, track ID, codec declaration, timescale, dimensions, sample count, and every sample’s timing/sync/size/digest tuple.

Changing one composition offset changes the track-manifest identity even if sample payload bytes do not change. Changing metadata outside the track can change the container digest while preserving sample payload digests. Those are useful distinctions for caches and investigations.

Step 5: cross a decoder boundary honestly

Demux output is not a pixel buffer. A decoder adapter should accept an exact track release and sample sequence, then emit:

source container and track-manifest digests;
decoder implementation, version, build, codec configuration, and hardware path;
associated sample ID or sample span;
PTS and duration under a named timeline;
coded and visible dimensions;
pixel format, bit depth, strides, range, primaries, transfer, and matrix;
rotation/flip/aperture decisions;
decoded plane digests;
concealment, corruption, and dropped-frame observations.

This chapter’s DecodedLumaFrame is deliberately smaller: sample ID, PTS, duration, width, height, bounded luma bytes, and a domain-separated digest that binds all of them. It is sufficient for the cut/motion laboratory, not a complete archival decoded-frame format.

Do not compute luma by treating compressed bytes as pixels. Do not interpret Y as display brightness without retaining range and transfer metadata. For HDR and wide-gamut footage, scene-linear or perceptually appropriate analysis may be necessary.

Step 6: separate cut evidence from motion evidence

A frame difference can rise because of:

object motion;
camera pan, tilt, zoom, or shake;
a hard cut;
a dissolve or fade;
exposure or white-balance change;
flash photography;
encoder corruption or concealment;
noise, grain, rain, or display animation.

One scalar cannot identify all of these causes.

Histogram cut score

The companion bins luma into 16 bins. For histograms H₀ and H₁ over N pixels:

cut_score = Σ |H₁[b] - H₀[b]| / (2N)

The result is scaled to 0–10,000 basis points. Dividing by 2N gives 10,000 when the distributions have disjoint support. This is insensitive to spatial arrangement: moving a textured image can keep nearly the same histogram, while a black-to-white cut scores maximally.

That is helpful, but not sufficient. Two different scenes with similar histograms can score low. A global exposure jump can score high. Thresholds must be calibrated on the product’s footage and reviewed by slice: animation, screen recordings, night video, sports, talking heads, and generated media behave differently.

FFmpeg’s scdet filter similarly exposes a per-frame change metric and scene score with a configurable threshold. Its documentation recommends a range for that implementation; do not transplant the number to a different metric.

Uncompensated change

The mean absolute luma difference answers, “How much did pixels change at the same coordinates?” It is useful as temporal activity, but camera translation makes it large even if the scene is unchanged.

Bounded block motion

The companion partitions the interior into textured blocks. For each current block, it searches a bounded radius in the previous frame and chooses the previous block with the smallest sum of absolute differences (SAD). Ties prefer lower Manhattan displacement and then stable signed coordinates. The reported motion vector points from the matching previous block to the current block.

The policy bounds:

block size;
search radius;
minimum luma span needed to call a block textured;
cut threshold;
total pixel comparisons.

The comparison cost is approximately:

blocks × (2 × radius + 1)² × block_width × block_height

The implementation charges this cost before searching each block and fails when the budget would be exceeded. A caller cannot turn a “simple” analysis endpoint into unbounded quadratic work by requesting a huge radius.

After compensation, the residual indicates how well translation explained the new block. Low residual plus coherent vectors supports a translation interpretation. High residual, random vectors, or few textured blocks weakens it.

This is an educational deterministic estimator, not dense optical flow. Occlusion, rotation, scale, rolling shutter, deformation, compression, and large motion require stronger algorithms.

Step 7: sample by time coverage, not frame count

“Take every 30th frame” has no stable time meaning under VFR. “Take one frame per second” still needs an exact selection rule.

For each target tick t, the companion finds a frame whose validated presentation interval contains it:

PTS <= t < PTS + duration

If no interval contains t, it records t in uncovered_target_ticks. It does not choose the nearest frame, because that would erase evidence of a gap.

The plan can also include:

first presentation frame;
last presentation frame;
every sync sample;
the first frame after every validated cut.

Reasons merge in a set, so one frame can satisfy several policies without duplication. The final selection is sorted by presentation time. A hard max_targets bounds timeline iteration and a hard max_selected_frames rejects—not truncates—an overfull important set.

Why keyframes are not a summary

A sync sample supports random access. It can land mid-shot, on a blur, or on a visually empty moment chosen for encoder rate control. Conversely, a semantically important event can occur between sync samples. Use sync status to plan decode entry and seek cost; combine it with temporal, cut, motion, semantic, and diversity policies for evidence selection.

FFmpeg’s thumbnail filter selects a representative frame from batches and warns that larger batches increase memory. That is a useful alternative for visual representativeness, but frame-batch sampling is not a substitute for exact timeline coverage.

Decode-efficient sampling

Selecting a non-sync sample may require decoding from the preceding random-access point through the target. A production sampler should:

plan target presentation intervals;
map targets to sample IDs;
find the required preceding sync samples;
merge overlapping decode spans;
decode spans once in DTS order;
retain only requested presentation frames;
record decoded-but-discarded work for cost analysis.

Seeking independently for every target repeats GOP decode and can make sparse sampling more expensive than a sequential pass.

Alternatives and where they belong

Pure-Rust MP4 parser

The selected mp4 crate keeps the chapter portable and makes sample-table logic visible. Its documented surface is small and it is not a universal adversarial-media sandbox. Keep exact version identity and narrow the accepted formats.

FFmpeg/libavformat and libavcodec

FFmpeg supports many containers/codecs and exposes packet/frame probing. ffprobe -show_packets and -show_frames are useful independent diagnostics. Native bindings expand format coverage but introduce ABI, build, licensing, unsafe-boundary, and deployment work. A supervised worker process often gives a better failure boundary than loading every codec into an API process.

FFmpeg can export codec motion vectors for supported codecs; its AVMotionVector fields include source direction, block size, source/destination positions, scaled motion components, and motion scale. Codec vectors optimize compression and are not automatically ground-truth object motion.

GStreamer

GStreamer is attractive for live pipelines, hardware elements, clock synchronization, and backpressure. It adds plugin discovery, capability negotiation, native deployment, and pipeline state complexity. Treat the pipeline description and plugin inventory as release identity.

WebCodecs

WebCodecs provides browser decoder primitives and explicit VideoFrame timing/resource semantics. Browser support, codec availability, hardware paths, cross-origin restrictions, and implementation versions remain part of reproducibility.

Platform frameworks

VideoToolbox/AVFoundation, Media Foundation, and Android MediaCodec can provide efficient hardware decode. They also create platform-specific color, timing, surface, and lifecycle behavior. Keep the portable domain contract above platform adapters.

Security and failure containment

Media parsers and codecs process attacker-controlled counts, offsets, lengths, entropy streams, and metadata. Production ingestion should layer:

upload and object-size limits before parsing;
magic-byte/container allowlists;
track/sample/dimension/duration limits;
checked arithmetic and allocation preflight;
CPU, memory, file-descriptor, and wall-time budgets;
cancellation and worker termination;
sandboxed low-privilege processes for broad native codecs;
read-only input and isolated scratch storage;
no network access in decoder workers unless explicitly required;
exact parser/decoder release identity;
quarantine and reproducible failure artifacts;
dependency advisories and license review;
fuzzing of wrapper and parser boundaries.

Never log raw frame bytes, private thumbnails, signed URLs, or full user paths. Structured diagnostics need container digest prefix, track/sample ID, stage, release, bounded dimensions, and typed failure classification.

Failure investigation laboratory

Thumbnail is consistently early

Inspect the target clock, timescale, conversion rounding, edit mapping, PTS/DTS distinction, and whether an API used frame index. Compare sample tables with an independent ffprobe report.

B-frames appear out of order

Confirm processing is sorted by PTS for presentation analysis and by DTS for decoder submission. Never repair this by sorting encoded bytes and sending them to the decoder in PTS order.

VFR clip drifts over minutes

Search for rounded frame-duration accumulation or nominal-FPS arithmetic. Recompute targets against original integer timestamps.

Cut detector fires on camera flashes

Inspect histogram score, uncompensated change, compensated residual, duration of the change, and recovery on the next frame. Add a temporal flash rule or a learned calibrated detector; do not only raise the global threshold.

Motion vectors are random at cuts

That is expected: block correspondence is undefined across unrelated scenes. Gate motion interpretation when cut evidence is high, and keep the raw metrics for audit.

Motion is zero on a blank wall

Low-texture blocks cannot provide unique correspondence. The implementation skips them. “No measurable motion” is not “camera stationary.”

Sampling misses a short event

Uniform interval coverage guarantees only coverage at its target grid. Add cut, motion-peak, audio-event, transcript, detector, and uncertainty-driven candidates, then apply diversity and budget selection.

Sample parser uses too much memory

Verify stsz preflight occurred before payload reads, lower file/sample limits, and inspect parser metadata allocations. Move broad parsing into a memory-limited worker. A wrapper bound cannot fix a parser that allocates from malicious nested counts before exposing them.

Tests as executable claims

The module has thirteen focused tests. They prove:

a real MP4 round-trip preserves DTS order, PTS order, sync samples, VFR duration, and bounds;
wrong magic and oversized input fail;
multiple video tracks require explicit selection;
missing requested tracks fail;
sample count and preflight payload size limits fail;
rational tick conversion rounds outward for positive and negative values;
decoded luma identity binds pixels, geometry, sample, and time;
one-pixel translation is recovered with zero compensated residual;
black-to-white distribution change produces a maximal cut score without invented motion blocks;
analysis comparison cost fails closed;
sampler reasons merge deterministically;
presentation gaps are reported;
important-frame sets are rejected rather than silently truncated;
invalid policies fail before media work.

The example adds a fourteenth focused test over the complete demux→analysis→sampling report.

Run the narrow gates:

cargo test -p mosaic-evidence video::tests
cargo test -p mosaic-evidence --example video_pipeline
cargo clippy -p mosaic-evidence --all-targets --all-features -- -D warnings

Then run the complete companion gate:

sh scripts/ci.sh
cargo deny check

Exercises

Recall

What is the difference between DTS and PTS?
Why is a sync sample not necessarily a representative frame?
What does a half-open presentation interval prevent at an exact boundary?
Which MP4 table should be inspected before allocating a variable-size sample?
Why can average FPS not locate the frame visible at 2.5 seconds?
What does zero block motion mean in a textureless region?

Implementation

Add a presentation_gaps() method returning exact tick ranges and covering microsecond views.
Add a decode-span planner that maps selected samples to merged preceding-sync-sample spans.
Add a policy option to reject gaps larger than a configured number of ticks.
Add a 32-bin histogram release and prove the release changes artifact identity.
Add per-block vector output behind a strict block-count limit.
Add a dissolve detector using several consecutive histogram scores rather than one threshold.
Cross-check the generated fixture with ffprobe JSON in an optional integration test.
Create a malformed fixture whose stsz entry exceeds the file and verify preflight rejects it before read_sample.

Design

Design an edit-list mapping artifact that preserves raw media time and authored movie time.
Design a fragmented-MP4 adapter for partial uploads and discontinuous fragments.
Define a video decoder capability trait without pretending every backend supports the same pixel formats, hardware surfaces, error concealment, or deterministic output.
Create a sampling objective that balances time coverage, cuts, motion, semantic novelty, faces, text, and a hard decode budget.
Threat-model a public video-analysis API using a native decoder worker.
Design a human-review UI that displays exact cited intervals, neighboring frames, audio, cut score, motion evidence, and selection reason.

Solution guidance

A gap detector sorts by PTS, carries the prior exclusive end, and emits [prior_end, next_pts) when next_pts > prior_end. Do not convert to milliseconds until display.
For each selected sample, walk backward in presentation intent but locate the preceding sync point in decode dependency order. Merge overlapping inclusive sample-ID spans.
A gap policy belongs after validated intervals, not in the container parser. Preserve both the rejection and observed gap.
A metric release must bind bins, color/luma preparation, resize, threshold convention, and rounding. Comparing scores across releases requires an explicit compatibility study.
Dense block output can exceed the frame itself. Bound blocks, coordinates, vector components, and serialized bytes.
A dissolve produces sustained moderate changes; a flash often rises and reverses. Use a bounded temporal window and calibrate against labelled fixtures.
An independent probe is a differential oracle, not absolute truth. Pin its build and compare exact track/sample fields with documented tolerances.
Mutate sample-table metadata, not only truncate payload bytes. Run the parser in the same sandbox expected for untrusted fixtures.

Check your understanding

If DTS is monotonic but PTS order is 1,3,2,4, which order should a decoder receive and which order should a scene detector analyze?
A target tick equals one frame’s exclusive end and the next frame’s inclusive start. Which frame owns it?
If a one-second sample follows thirty 33 ms samples, is “30 FPS” sufficient to sample it?
If payload digests match but the composition offsets change, which identity must change?
If histogram score is low and uncompensated difference is high but compensated residual is zero, what explanation is supported?
Why should an over-budget set of cut frames fail instead of taking the first N?
Which claims in this chapter remain invalid until a real decoder adapter is implemented?

Completion checklist

Authoritative references

What comes next

This chapter keeps one video track internally coherent. The next chapter builds a shared alignment model across video PTS, audio sample clocks, text spans, image regions, edits, generated media, and wall time. Only after those mappings are explicit can Mosaic claim that a transcript phrase, sound event, visual region, and cited video interval refer to the same real moment.