Timeline and Geometry Alignment Across Modalities

“The transcript and the face box both occur at 01:00” is not yet an alignment claim.

Which minute? The audio decoder counts sample frames. The video track counts presentation ticks. WebVTT cues use media-relative time. A capture device may also expose a monotonic hardware clock, while a live transport periodically relates its own timestamp counter to a wall clock. The vision model receives a cropped, resized, and padded tensor that no longer shares coordinates with the decoded frame. Even the phrase “the same frame” is ambiguous when one system means coded pixels, another means the visible rectangle after rotation, and a third means the model’s 224 × 224 input.

Alignment is therefore not a decimal timestamp or a copied bounding box. It is a versioned, testable transformation between named coordinate systems.

This chapter builds two such transformations:

a piecewise rational clock map from each source clock into one canonical program clock; and
an exact affine geometry map from oriented source pixel edges into model tensor pixel edges.

The implementation maps interval starts down and exclusive ends up, so derived ranges cover the source claim. It never interpolates across a missing clock segment. It composes crop, resize, and letterbox operations without floating-point drift. It can invert a nonsingular transform to back-project a model result into the source image. Both map types have stable SHA-256 identities, so a citation can name not only its artifact and locator but also the transformation that made the locator meaningful.

The result is intentionally narrower than a full synchronization estimator. It executes a declared calibration exactly; it does not infer anchors from arbitrary media and does not invent confidence for them. A production system must store calibration evidence, residual error, and uncertainty alongside this deterministic map. That boundary is a feature: arithmetic certainty must not be confused with measurement certainty.

Learning objectives

By the end you will be able to:

name audio sample, video presentation, subtitle, transport, monotonic, wall, and program clocks without treating them as interchangeable;
explain why a timescale defines units but not an origin or synchronization relationship;
derive offset and rate from two anchors and recognize why one anchor cannot measure drift;
calculate drift in parts per million and decide when a single affine clock map is insufficient;
represent splices, pauses, capture restarts, and rate changes as ordered piecewise segments;
map half-open time intervals with exact integer arithmetic and outward rounding;
preserve unmapped source ranges rather than stretching a nearby segment across a discontinuity;
intersect audio and video evidence only after both are mapped into the same target clock;
distinguish coded, visible, oriented, cropped, resized, padded, tensor, and display spaces;
reason in pixel-edge coordinates and map all four corners of a half-open rectangle;
compose translation, scale, rotation, reflection, and padding into one rational affine map;
invert a nonsingular geometry transform and conservatively back-project a model box;
attach stable identity to temporal and spatial mappings;
calibrate with observable signals rather than circular model agreement;
propagate measurement uncertainty separately from deterministic rounding;
test boundary, drift, gap, rotation, inverse, overflow, and identity behavior;
investigate audiovisual “off by one frame” failures without guessing at a global offset.

Prerequisites and dependency order

Read these chapters first:

Evidence envelopes define the immutable acquired artifacts.
Provenance graphs make transformations and observations first-class.
Text normalization maps normalized ranges back to original bytes.
Image decoding establishes oriented pixels and bounded half-open regions.
Audio clocks retains sample-frame coordinates across decoding and resampling.
Video timing retains PTS, duration, variable-frame-rate intervals, cuts, and sampled-frame identity.

Alignment comes after modality-specific normalization because a map needs stable source coordinates. It comes before multimodal retrieval because retrieval candidates need comparable time ranges and source-restorable geometry. It also precedes Cutroom: an edit decision is unsafe when speech, frame, and generated overlay coordinates have been joined by an unexplained constant.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example alignment_pipeline

The example produces one JSON report containing:

a two-segment map from drifting 48 kHz audio source frames to program microseconds;
a map from 90 kHz video presentation ticks to the same program clock;
a speech interval that crosses the audio rate boundary;
a video frame interval that overlaps the speech;
two exact aligned program windows;
a geometry map from a cropped 1920 × 1080 oriented frame to a 224 × 224 tensor;
the model-space location of a person;
the exact source region recovered by the inverse transform;
stable identities for every map.

The audio fixture observes 48,000 frames during the first program second and 48,048 frames during the second. Its second rate is 1,000 ppm faster than nominal:

(48,048 - 48,000) / 48,000 × 1,000,000 = 1,000 ppm

The speech range [47,900, 48,100) source frames crosses that change. It becomes two program ranges rather than one silently averaged range:

[997,916, 1,000,000) microseconds
[1,000,000, 1,002,082) microseconds

The video frame [89,550, 90,090) at a 90 kHz timescale becomes:

[995,000, 1,001,000) microseconds

Their intersection is:

[997,916, 1,000,000)
[1,000,000, 1,001,000)

The adjacent results remain separate because they came through different calibrated audio segments. A caller may coalesce them for display, but the evidence layer retains the boundary.

For geometry, the example keeps source X coordinates [240, 1680), scales the resulting 1440 × 1080 crop by 7/45 to 224 × 168, and adds 28 pixels of top padding. A source person region (x=690, y=180, width=450, height=720) becomes (70, 56, 70, 112) in the tensor. Applying the exact inverse recovers the original region.

Coordinate systems are types, even when the language says `i64`

A number without a coordinate system is incomplete data.

These values may all be 1_000, but they mean different things:

Value	Coordinate system	Possible interpretation
`1_000`	audio sample frames at 48 kHz	about 20.833 ms from the audio origin
`1_000`	video PTS ticks at 90 kHz	about 11.111 ms from the track origin
`1_000`	program microseconds	1 ms from the chosen program origin
`1_000`	Unix milliseconds	an instant in January 1970
`1_000`	tensor X pixel edges	invalid for a `224`-wide tensor

The companion makes clock IDs, timescales, geometry-space IDs, dimensions, and map identities explicit. Rust cannot prevent a caller from passing the wrong TimeRangeTicks to a map because the current API uses one range type, but the map report always retains its source and target map identity. A larger API can add newtypes such as AudioFrameRange and ProgramTimeRange; the serialized schema must still carry the clock names because newtypes disappear on the wire.

Do not use a unit name as if it were a complete clock name. microseconds states a scale, not an origin. Prefer program_microseconds, camera_a_pts_90khz, or a structured clock identifier tied to the artifact and track.

Temporal alignment from first principles

Timescale, origin, rate, and continuity are different claims

A clock mapping needs more than “both are seconds.”

Timescale says how many ticks represent one nominal second.
Origin says what source tick corresponds to what target tick.
Rate says how a source span corresponds to a target span.
Continuity says whether mapping between anchors is valid or a restart/splice creates a gap.
Uncertainty says how far the real event may be from the mapped coordinate.

Two 48 kHz devices can start at different moments and drift differently. Two tracks in one MP4 can have different timescales but a shared edit timeline. Two RTP media sessions deliberately have separate timestamp spaces. RFC 3550 explains that synchronized audio/video playback is achieved through timing information in RTCP for both sessions, not by directly comparing their RTP timestamps. See RTP and RTCP synchronization and its audio/video session example.

One anchor fixes offset; two anchors reveal rate

Suppose a clap occurs at audio frame 96,000 and program time 2,000,000 µs. That pair fixes one point:

audio 96,000 ↔ program 2,000,000

Infinitely many slopes pass through it. Assuming the nominal 48 kHz rate is an additional claim, not a consequence of the clap.

A later trusted event gives a second pair:

audio 576,480 ↔ program 12,000,000

Now the observed span is 480,480 source frames per 10,000,000 target microseconds, or 48,048 frames per second. The affine mapping within those anchors is:

target = target₀
       + (source - source₀)
       × (target₁ - target₀)
       / (source₁ - source₀)

No floating point is necessary. The companion stores each pair of anchor endpoints as one ClockSegment.

Why the map is piecewise

A single global line is attractive and often false. Create a new segment when:

a device restarts its monotonic counter;
a recorder pauses and resumes;
an edit removes or inserts material;
an RTP timestamp wraps or a sender changes clock behavior;
capture rate changes after thermal or power management;
a resampler switches ratio;
an anchor residual exceeds the error budget;
a live source is reconnected;
a clip is placed at a new program offset.

Each segment must increase in source and target time. Segments must be ordered and non-overlapping in both domains. Source or target gaps are legal and meaningful. An overlap is ambiguous and fails construction.

let audio_map = PiecewiseClockMap::new(
    "audio_source_frames",
    48_000,
    "program_microseconds",
    1_000_000,
    vec![
        ClockSegment::new(0, 48_000, 0, 1_000_000)?,
        ClockSegment::new(48_000, 96_048, 1_000_000, 2_000_000)?,
    ],
)?;

The declared source_timescale remains useful metadata, but the segment endpoint spans determine the actual mapping. This permits measured drift instead of forcing every segment to nominal rate.

Half-open ranges and outward rounding

An evidence interval is [start, end). It contains the start and excludes the end. Adjacent intervals can meet without both claiming the same instant.

For a source boundary s inside one segment, let:

S = source_end - source_start
T = target_end - target_start

The exact rational target boundary is:

target_start + (s - source_start) × T / S

When target ticks cannot represent the rational value, the mapped interval starts with floor and ends with ceiling. At a three-tick source timescale mapped to one million target ticks, source [1,2) covers:

[333,333⅓, 666,666⅔) exact
[333,333, 666,667) covering integer range

This may add less than one target tick on each side. It does not claim that the physical event grew; it states the smallest integer target interval guaranteed to cover the declared rational interval. The implementation uses checked i128 intermediates and Euclidean division, including for signed coordinates. Overflow returns a typed error.

Never repeatedly add a rounded frame duration. Map each boundary from its original integer coordinate. Repeated addition accumulates error.

Gaps are outputs, not exceptions to hide

Consider source coverage [0, 90,000) and [180,000, 270,000). The middle source range is absent. Mapping [45,000, 225,000) must return:

a mapped piece from the first segment;
unmapped source [90,000, 180,000);
a mapped piece from the second segment.

Stretching the first segment through the gap creates evidence for media that does not exist. Assigning the gap to the nearest frame creates a false citation. map_range_covering therefore returns a TimeMappingReport with both mapped and unmapped_source.

Callers choose policy:

retrieval can exclude unmapped pieces;
a player can show “alignment unavailable”;
an editor can require manual calibration;
an approximate UI can display a dashed uncertainty band;
a strict export can reject the operation.

The evidence layer does not make that product decision by truncating silently.

Join only inside a canonical clock

Do not compare an audio sample range directly with a video PTS range. Map both into a named canonical clock, then intersect the ordered target ranges:

let speech_program_ranges: Vec<_> = speech_mapping
    .mapped
    .iter()
    .map(|piece| piece.target)
    .collect();
let frame_program_ranges: Vec<_> = video_frame_mapping
    .mapped
    .iter()
    .map(|piece| piece.target)
    .collect();

let overlap = intersect_time_ranges(
    &speech_program_ranges,
    &frame_program_ranges,
)?;

The two-pointer intersection is linear in the number of ordered pieces and never bridges a gap. The canonical clock can be program microseconds, composition ticks, or another rational domain. Choose one stable enough for the workflow; do not force UTC into an offline edit when a monotonic program timeline is sufficient.

Wall clocks are useful and dangerous

Wall time answers questions such as “which camera observation coincided with this sensor event?” It can also step, slew, include leap-second behavior, or arrive through a mapping with network jitter. A smooth playout clock and civil UTC are not the same abstraction.

RFC 7164 documents complications in RTP-to-NTP mapping around leap seconds and recommends a smooth time base for smooth media playout. See RFC 7164: RTP and leap seconds. RFC 7273 specifies signaling for reference clock and media clock sources: RTP Clock Source Signalling.

Store:

the reference-clock type and identifier;
observed source/reference anchor pairs;
capture method and software release;
monotonic observation time;
uncertainty and residuals;
clock discontinuity events;
wrap-extension state for bounded counters.

Do not call SystemTime::now() once for audio and once for video and treat the difference as media alignment. Scheduling latency is not a calibrated clock map.

Subtitle and transcript clocks

WebVTT cue offsets are interpreted relative to media playback position. That does not make a cue’s text byte offsets or ASR sample indices interchangeable with video PTS. Preserve:

original cue timing and cue identity;
normalized text range and its mapping to source bytes;
the cue clock’s map to the canonical program timeline.

See the WebVTT cue timing model.

An ASR system may emit timestamps quantized to 20 ms feature frames while the source audio uses 48 kHz samples. Keep both. A word locator can contain the ASR interval plus its source-audio covering interval; the latter then maps to program time.

Measurement uncertainty is not rounding error

The companion map is exact given its declared anchors. It does not prove the anchors are exact.

These are separate errors:

integer covering adds less than one target tick per boundary;
a clap detector may localize audio within ±3 ms;
a flash detector may localize only to one VFR frame;
transport reports may include network and scheduling error;
hardware clocks may drift between anchors;
a fitted segment has residual error against held-out anchors.

A production schema should attach at least max_error_before and max_error_after to each mapped boundary or a calibrated uncertainty model to each segment. Conservative temporal overlap expands each range by its uncertainty before testing possible overlap. Definite overlap instead contracts each range; if the contracted intervals do not meet, only possible overlap is known.

Do not add all errors blindly when correlations are known, and do not use root-sum-square unless the statistical assumptions are defensible. For safety-critical citations, a documented worst-case bound is easier to audit than an unexplained confidence score.

The current mosaic-clock-alignment-v1+piecewise-rational release deliberately omits estimated uncertainty. Its artifacts are suitable for deterministic calibrated fixtures and exact transform replay. Before using it to make real-world simultaneity claims, extend the schema, bump the release, include uncertainty fields in the map digest, and add calibration residual tests.

Geometry alignment from first principles

Name every image space

A useful vision pipeline can contain all of these:

coded plane — decoder allocation, possibly including padding;
visible rectangle — the intended visible pixels inside coded storage;
oriented image — rotation and reflection applied;
crop — a selected source domain;
resized image — interpolation changes dimensions;
letterboxed or center-cropped image — aspect-ratio policy changes placement;
tensor — channel order, normalization, and batching are added;
feature map — model stride creates another spatial grid;
model output — boxes may be pixels, normalized values, centers, or corners;
display — CSS or UI scaling introduces another transform.

WebCodecs exposes coded size, visible rectangle, display size, rotation, and flip separately on a VideoFrame; see the WebCodecs VideoFrame interface. Collapsing these fields into “width and height” loses the path back to source evidence.

Use identifiers such as video_oriented_pixels, vision_tensor_pixels, and detector_stride_16_cells. Include dimensions. If model output uses normalized coordinates, record whether normalization divides by width, width minus one, or another convention. Those choices are observably different at the edge.

Pixel edges versus pixel centers

The companion represents an integer rectangle:

(x, y, width, height)

as a half-open region between pixel edges:

[x, x + width) × [y, y + height)

Pixel (x, y) has its center at (x + 0.5, y + 0.5) under the common edge-coordinate convention. Resize implementations often document their interpolation in center coordinates, while detection boxes are commonly easier to compose as edges. Mixing the conventions creates half-pixel shifts.

The geometry map answers a locator question: which integer target pixels cover the transformed source region? It maps all four edge corners, takes minima and maxima, floors the left/top, and ceils the right/bottom. This remains conservative under rotation and shear.

Interpolation of pixel values is a separate algorithm. OpenCV, for example, describes geometric image transformations through inverse mapping and interpolation in its geometric image transformation documentation. An evidence locator must record both the coordinate transform and the pixel-resampling release if the derived raster itself matters.

One rational affine transform

The map uses:

x' = (m00·x + m01·y + m02) / d
y' = (m10·x + m11·y + m12) / d

with integer coefficients and positive denominator d. This represents translation, anisotropic scale, rotation, reflection, and shear. Perspective warps require a homography and lens distortion requires another model; do not force them into an affine claim.

The example’s crop, scale, and letterbox are:

crop:      x₁ = x - 240, y₁ = y
scale:     x₂ = 7x₁/45,  y₂ = 7y₁/45
letterbox: x₃ = x₂,      y₃ = y₂ + 28

Combined:

x' = (7x - 1680) / 45
y' = (7y + 1260) / 45

and in Rust:

let transform = Affine2D::new(
    7, 0, -1_680,
    0, 7,  1_260,
    45,
)?;

The constructor rejects a zero denominator and a singular linear part. It reduces coefficients by their greatest common divisor so equivalent expressions receive a canonical representation. Composition uses checked i128 intermediates and returns after(self(point)), making operation order explicit:

let crop = Affine2D::new(1, 0, -240, 0, 1, 0, 1)?;
let resize_and_pad = Affine2D::new(7, 0, 0, 0, 7, 1_260, 45)?;
let combined = crop.then(resize_and_pad)?;

Matrix multiplication is not commutative. Padding then scaling is different from scaling then padding. Test the expected coefficients and at least one known region.

Source domain matters

The source space is the entire 1920 × 1080 oriented frame, but the transform is valid only for the crop:

let map = GeometryMap::new(
    GeometrySpace::new("video_oriented_pixels", 1_920, 1_080)?,
    PixelRegion {
        x: 240,
        y: 0,
        width: 1_440,
        height: 1_080,
    },
    GeometrySpace::new("vision_tensor_pixels", 224, 224)?,
    transform,
)?;

A region in the discarded left 240 pixels must fail. Applying the formula would produce a negative target coordinate, but the deeper issue is semantic: those pixels never entered the model. GeometryMap::map_region_covering validates that every source region lies inside source_domain.

Construction also maps the complete domain and checks that its covering target region fits inside the declared target dimensions. Bad scale, sign, padding, or dimension metadata therefore fails before observations are processed.

Back-project model outputs

A model box in tensor space is useful for citation only when it can return to a source space.

For a nonsingular affine map:

x' = Ax + t

the inverse is:

x = A⁻¹(x' - t)

The companion computes the exact rational inverse and applies the same four-corner covering rule:

let source_region = map
    .transform
    .inverse()?
    .map_region_covering(
        tensor_region,
        map.target.width,
        map.target.height,
        map.source.width,
        map.source.height,
    )?;

Round trips are not always exact. If a source region maps to fractional target edges, outward rounding enlarges the tensor region; back-projecting that covering box can enlarge the source region again. That is conservative and expected. The fixture was chosen so its edges transform exactly, which makes an equality assertion appropriate. Add non-exact property tests that assert the round-trip result contains the original, not that it always equals it.

Rotations require all corners

Mapping only top-left and bottom-right works for axis-aligned positive scaling, then fails for a quarter turn, reflection, or shear. A 90-degree transform can move top-right to the new bottom-right and bottom-left to the new top-left.

The companion tests:

x' = 100 - y
y' = x

and maps source (10, 20, 30, 40) to target (40, 10, 40, 30). All four corners are evaluated before the enclosing integer region is constructed.

Geometry identity is part of observation identity

Changing top padding from 28 pixels to 28⅟₄ pixels changes where the model looked. The map digest therefore includes:

source space ID and dimensions;
exact source domain;
target space ID and dimensions;
all six affine coefficients;
denominator;
a domain-separated schema prefix.

Changing any coefficient changes map_sha256. A vision observation should reference:

acquired image or video-frame identity;
source locator;
geometry-map identity;
preprocessor/model release;
model-space output;
back-projected source coverage.

Do not hash debug JSON as a stable protocol. The companion hashes a fixed field order with length-prefixed strings and big-endian integers.

Calibration: obtain anchors without circular reasoning

The arithmetic is only as trustworthy as the anchors.

Designed calibration signals

Prefer signals deliberately visible in multiple modalities:

a clapper produces an audio impulse and a visible closure;
a flash plus electronic pulse joins camera, microphone, and device clock;
timecode or genlock supplies a hardware reference;
a slate displays an identifier while speaking or beeping it;
a sync LED driven by the same circuit as an audio pulse gives repeated anchors.

Repeated signals measure both offset and drift. Place them across the recording, not only at the start.

Natural signals

When designed signals are unavailable, cross-correlate physically related events:

lip closure and plosive energy;
impact motion and transient audio;
musical onset and hand movement;
screen transition and captured system sound.

Keep candidate matches, correlation score, ambiguity, chosen window, filtering parameters, and human confirmation. Natural events can repeat. A periodic beat may create several equally strong offsets.

Do not use a multimodal model’s generated statement—“these seem synchronized”—as the sole anchor and then grade the same model on aligned evidence. That creates circular validation. Models can propose anchors; independent signals and held-out checks must validate them.

Fit, segment, and validate

A defensible workflow is:

collect anchor pairs with per-anchor localization bounds;
reserve some anchors for validation;
fit an affine segment robustly;
inspect signed residuals over time;
split at known discontinuities or systematic residual changes;
refit each segment;
reject segments with too few independent anchors;
store fit release, anchor identities, residual summary, and uncertainty;
validate on held-out anchors;
freeze and digest the accepted map.

An excellent mean residual can hide one unacceptable local error. Report maximum absolute, percentiles, and residual-versus-time plots. For lip-sync, evaluate signed error because audio leading video and video leading audio have different perceptual consequences.

Failure investigation: the subtitle points at the previous shot

Suppose a retrieved quote at 00:12.040 displays a face from the previous shot.

Do not start by adding 40 ms globally. Work from identities and boundaries:

Reproduce the exact run. Load artifact, track, decoder, frame-sampler, transcript, clock-map, geometry-map, and retrieval release identities.
Inspect original coordinates. Is 12.040 a video PTS, program time, ASR time, or UI string?
Inspect intervals, not points. Compare the word’s source-audio range and the frame’s full presentation interval.
Check segment selection. Did the word cross a drift boundary? Was a gap silently removed by a higher layer?
Check timebase conversion. Look for rounded accumulated frame durations, integer truncation, milliseconds mistaken for microseconds, or a nominal-FPS formula.
Check decode versus presentation order. A DTS sorted as PTS can select a neighboring frame.
Check edit placement. Container media time may not equal program time after an edit list.
Check uncertainty. The current deterministic map may be replaying an imprecise anchor exactly.
Check geometry. If the time is right but the highlighted face is wrong, verify visible rectangle, orientation, crop, letterbox, and inverse map.
Add the smallest failing fixture. Preserve the discontinuity or fractional boundary that exposed the bug.

Only change the calibration when evidence shows the calibration is wrong. A UI compensation constant can make one sample look correct while worsening the rest.

Security, bounds, and adversarial cases

Alignment inputs are untrusted metadata even after media decoding.

The companion bounds identifier length, timescales, segment count, and geometry dimensions. Arithmetic is checked. Empty intervals, overlapping segments, singular transforms, invalid crops, and mapped regions outside target bounds fail closed.

Production limits should also cover:

anchor count and serialized map size;
total mapped pieces created by range splitting;
decimal or rational denominator magnitude;
recursive transform-chain depth;
calibration search duration and lag window;
cross-correlation FFT allocation;
retained residual samples;
maximum uncertainty;
permitted clock-source types;
model box count before back-projection.

Adversarial cases include a billion tiny clock segments, extreme coefficients that overflow composition, near-singular transforms that magnify localization error, NaN model coordinates, negative normalized boxes, deceptive repeated sync signals, metadata claiming a false rotation, and a transform chain that alternates crop and upscale until resource limits fail.

If calibration affects access control, billing, medical interpretation, or legal evidence, require signed map artifacts, separation of duties, and an audit log. A stable digest detects change; it does not tell you whether the signer was authorized or the anchors were honest.

Performance and architecture

Mapping n ranges through m segments need not scan from the beginning for every range. The companion’s simple per-range scan is transparent and sufficient for the fixture. A production batch path should:

sort and validate ranges once;
advance a shared segment cursor;
binary-search the first relevant segment for random queries;
preallocate with bounded estimates;
retain piece provenance without copying large payloads;
cache immutable maps by digest;
keep clocks and geometry independent from decoded pixel/audio buffers.

Intersection of two ordered range sets is O(n + m). Affine region mapping is constant work: four points and checked extrema. The expensive operations are usually anchor discovery, decoding, resampling, correlation, and model inference—not applying a frozen map.

Do not optimize by converting exact ticks to f32. A few checked integer operations are negligible beside media processing, and they preserve reproducibility.

A practical service boundary is:

acquired artifact
  → bounded decoder/demuxer
  → source-clock + source-geometry observations
  → immutable calibration maps
  → canonical-time + source-restorable observations
  → retrieval / editing / generation harness

Store maps as content-addressed artifacts. Reference them; do not copy mutable “current offset” fields into every observation.

Test strategy

The chapter companion includes tests for:

a range crossing a measured drift boundary;
a source discontinuity returned as unmapped_source;
rejection of source or target overlap;
one-third tick outward rounding;
intersection that does not fill gaps;
exact crop/resize/letterbox mapping;
exact inverse back-projection;
composition equivalence;
a quarter-turn region using all four corners;
singular and out-of-bounds failure;
map digest change after an anchor or coefficient change;
the complete runnable report.

Extend it with:

Property tests

Mapping a nonempty subrange inside a valid segment returns nonempty target coverage.
Mapped starts never exceed exact rational starts; mapped ends never precede exact rational ends.
Ordered source subranges remain ordered within a positive-rate segment.
inverse(transform(point)) equals the rational point for representable test points.
Back-projecting a covering mapped region contains the original source region.
Composition equals sequential rational point application.

Fuzz tests

arbitrary serialized segment lists never panic or allocate beyond policy;
arbitrary affine coefficients either construct a bounded valid map or return a typed error;
arbitrary region dimensions never wrap;
malformed map identities and unknown fields fail deserialization.

Golden tests

Keep a small artifact with known clap/flash anchors, VFR frames, rotation, crop, and tensor boxes. Review the canonical program intervals and source back-projections. Golden files are appropriate for intentional schema changes, not as the only arithmetic oracle.

Metamorphic tests

shifting every source and anchor tick by the same amount leaves relative target coverage unchanged;
multiplying all segment endpoints by the same positive scale preserves mapped real time;
splitting one exact linear segment into two at a shared anchor preserves mapping;
composing with identity changes neither transform nor digest after canonicalization;
adding an unrelated gap does not change mappings before the gap.

Exercises

Exercise 1: derive a drift map

A recorder produces anchor pairs:

source frame 0       ↔ program 0 µs
source frame 240,120 ↔ program 5,000,000 µs

Calculate observed frames per second.
Calculate drift relative to 48 kHz in ppm.
Map source [48,000, 96,000) using outward rounding.
Explain why the declared 48 kHz timescale is still useful.

Exercise 2: preserve a pause

A camera maps [0, 90,000) to program [0, 1,000,000), pauses, then maps [90,000, 180,000) to [3,000,000, 4,000,000).

Construct the two segments.
Map the complete source range.
Identify whether there is a source gap, target gap, or both.
Decide how retrieval and an editor should represent the result.

Exercise 3: crop, scale, and pad

Crop source [100, 900) × [50, 650) from a 1000 × 700 image. Resize the 800 × 600 crop by 1/2, then add 20 target pixels of left padding and 10 of top padding.

Write the six affine numerators and denominator.
State the target dimensions if padding is symmetric.
Map source region (300, 150, 200, 100).
Invert the result.

Exercise 4: uncertainty

An audio interval maps to [1,000, 1,100) ms with ±8 ms calibration error. A video interval maps to [1,105, 1,140) ms with ±4 ms error.

Do the nominal intervals overlap?
Can the physical events possibly overlap?
Are they guaranteed to overlap?
What should a citation UI communicate?

Exercise 5: debug the offset

A lip-sync evaluation reports a median video lead of 33 ms, but residuals increase linearly from 0 ms at the start to 66 ms after one hour.

Why is a constant correction wrong?
Which observations are needed to distinguish clock drift from dropped-frame discontinuities?
Design a held-out validation set.
State which artifact identities must change after recalibration.

Solutions

Solution 1

Observed rate:

240,120 / 5 = 48,024 frames/s

Drift:

(48,024 - 48,000) / 48,000 × 1,000,000 = 500 ppm

Using the endpoint ratio, source [48,000, 96,000) maps to approximately [999,500.249..., 1,999,000.499...) µs, so integer covering is [999,500, 1,999,001). Compute from the original rational endpoints rather than a rounded frame duration. The 48 kHz timescale still declares the source tick unit and nominal media rate; measured anchors declare how that device clock relates to program time.

Solution 2

The source segments are contiguous at frame 90,000; there is no source gap. The target has a two-second gap [1,000,000, 3,000,000), representing the camera’s pause in program time. Mapping the complete source range returns two target pieces. Retrieval should not say the camera covers the pause. An editor can show an empty lane or a held frame only if the hold is an explicit edit, not an inferred property of the capture.

Solution 3

Crop, half-scale, then pad:

x' = (x - 100)/2 + 20 = (x - 60)/2
y' = (y - 50)/2  + 10 = (y - 30)/2

The affine coefficients are (1, 0, -60, 0, 1, -30, denominator=2). The resized content is 400 × 300; with symmetric padding it sits in 440 × 320. Source (300,150,200,100) maps exactly to (120,60,100,50). The inverse maps it back exactly because every edge is divisible under the chosen scale.

Solution 4

The nominal intervals do not overlap: there is a 5 ms gap. With uncertainty, audio may extend through 1,108 ms and video may begin at 1,101 ms, so physical overlap is possible. It is not guaranteed: audio may end at 1,092 ms while video begins at 1,109 ms. The UI should say “possible overlap within alignment uncertainty,” show the bounds, and link to the calibration artifact.

Solution 5

A constant changes intercept, while the linear residual trend indicates a rate mismatch. Collect repeated independent audiovisual anchors, original audio frames, video PTS/durations, decoder output order, capture-clock reports, and restart/drop events. Hold out anchors across the entire hour and around suspected discontinuities. Recalibration changes the clock-map digest and every derived observation or retrieval index whose identity includes it; acquired media digests remain unchanged.

Completion checklist

You are done when you can:

Further deductions

Several broader design rules follow from the implementation.

First, alignment belongs in provenance, not presentation state. A player offset slider can help a human inspect media, but it is not a reproducible evidence transform until its coordinate systems, calibration, and identity are frozen.

Second, canonical time should not erase source time. Program microseconds make cross-modal joins possible; source audio frames and video PTS make claims auditable and allow recalibration without re-decoding observations.

Third, geometry preprocessing is part of model semantics. The same weights applied after a different crop, resize kernel, padding color, or orientation are not the same observation procedure. Model identity alone is insufficient.

Fourth, coverage is safer than false precision. Outward-rounded intervals and enclosing boxes may be slightly larger, but they never pretend a fractionally covered source point vanished. Uncertainty makes this rule physical rather than merely arithmetic.

Fifth, map changes invalidate derived indexes. If an embedding chunk is assigned a program interval through clock-map digest A, recalibration to digest B can change temporal retrieval even when the embedding vector and source bytes remain identical. Index release identity must include the map.

Finally, multimodal reasoning needs a shared evidence plane, not a shared raw datatype. Audio frames, video ticks, text bytes, and tensor boxes should remain modality-specific. They cooperate through explicit transformations, canonical intersections, and citations that can travel back to their sources.

References

IETF, RFC 3550: RTP—A Transport Protocol for Real-Time Applications.
IETF, RFC 7164: RTP and Leap Seconds.
IETF, RFC 7273: RTP Clock Source Signalling.
IETF, RFC 7826: Real-Time Streaming Protocol Version 2.0.
W3C, Media Source Extensions.
W3C, WebCodecs.
W3C, WebVTT.
OpenCV, Geometric Image Transformations.
Rust standard library, i128::div_euclid.

The executable source of truth for this chapter is the local rust/rust-ai-engineering/crates/mosaic-evidence/src/alignment.rs companion. The workspace does not assume a public repository URL.