Audio — Codecs, Resampling, Channels, Timestamps, Speech, and Events

Audio is not “a file with a duration.”

It is a container carrying one or more packetized tracks. A selected track uses a codec. A decoder produces frames under a sample clock. Each frame contains one sample for every channel, and channel positions affect meaning. Lossy codecs can add delay, require pre-roll, conceal missing packets, and trim padding. Resampling creates a new clock and new samples. Speech recognition and acoustic-event models then add fallible observations whose timestamps must refer to one of those exact clocks.

If a system keeps only duration_ms, it cannot answer basic evidence questions:

Did sample 80 mean the 80th stereo frame or the 80th interleaved scalar?
Was the first decoded packet encoder delay or playable audio?
Did the transcript timestamp refer to encoded packets, source PCM, resampled mono, or wall time?
Was stereo averaged, selected, matrix-mixed, beamformed, or silently discarded?
Did the resampler remove frequencies above the new Nyquist limit?
Was “alarm” recognized in the same interval that the cited waveform displays?
Did a decoder error create a time gap, or was the damaged packet silently skipped?

This chapter implements a narrow, real boundary in mosaic-evidence. Symphonia 0.6 probes a byte buffer as RIFF/WAVE, selects an audio track, permits only PCM codecs, enables gapless trimming, and converts decoded samples to finite interleaved f32. The boundary enforces independent encoded, rate, channel, frame, sample, byte, packet, and observation bounds. Integer rational arithmetic maps half-open source-frame ranges to microsecond coverage. Explicit equal-weight mixing creates a separate mono artifact. Rubato 4 performs fixed-rate, anti-aliased FFT resampling and trims its startup delay through process_all.

The path is intentionally limited to WAVE plus PCM. It demonstrates the complete container→codec→playback-PCM contract without claiming that one decoder policy safely covers MP3, AAC, FLAC, Vorbis, Opus, multitrack broadcast material, or a live device. Those are extensions with new features, fixtures, delay rules, and security review.

Learning objectives

By the end you will be able to:

distinguish container, track, codec, packet, decoded frame, channel sample, and observation;
explain why extension, MIME type, container metadata, and decoder output are separate evidence;
bound encoded bytes, packets, sample rate, channels, frames, interleaved samples, and PCM memory;
prefer the first decoded buffer’s actual specification over an unverified declaration;
explain decoder delay, end padding, gapless output, seek pre-roll, and packet-loss concealment;
distinguish a sample frame from an interleaved scalar sample;
name channel positions and avoid treating channel count as channel meaning;
create an explicit downmix artifact and explain equal-weight cancellation risk;
use source sample frames as authoritative coordinates;
map sample ranges to a covering microsecond view without floating-point drift;
distinguish coverage rounding from an invertible timestamp conversion;
choose fixed-ratio FFT resampling for an offline clip and a drift-tracking resampler for live clocks;
preflight output size and retain resampler delay/identity;
hash PCM under a schema that binds rate, channels, layout, sample count, and sample bits;
keep ASR text, speaker labels, and acoustic events as ranged, versioned observations;
separate RMS, peak, loudness, speech quality, transcription quality, and event detection;
investigate timestamp shifts, channel loss, resampling artifacts, transcript drift, and decode failures;
design the next compressed-codec and streaming additions without weakening this boundary.

Prerequisites and dependency order

Read the first four multimodal chapters:

Evidence envelopes preserve the immutable encoded digest and label declared audio properties as unverified.
Provenance graphs bind each transcript, event, mix, and resample to an exact parent locator and transform release.
Text normalization handles derived ASR text without mistaking normalized-text offsets for audio coordinates.
Images established the pattern: encoded identity, bounded decode, normalized derived artifact, ranged observation, reproducible model input.

Also review blocking/media/accelerator scheduling and content addressing. Audio decode, resampling, and inference are CPU/memory work; they should not block an async executor worker.

Audio normalization precedes ASR, diarization, speaker recognition, audio embeddings, event detection, music analysis, audiovisual alignment, and video reasoning.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example audio_pipeline

The executable constructs a real 16-bit little-endian PCM WAVE fixture:

Property	Value
Encoded bytes	684
Sample rate	8,000 Hz
Channels	2, front-left + front-right
Frames	160
Interleaved scalar samples	320
Playback duration	20,000 µs
Speech interval	frames `[0,80)`, covering `[0,10,000)` µs
Alarm interval	frames `[80,160)`, covering `[10,000,20,000)` µs

The example then performs two explicit transformations:

decoded stereo PCM, 8 kHz, 160 frames
  └─ equal-weight channel average
      mono PCM, 8 kHz, 160 frames
        └─ fixed-ratio FFT resample
            mono PCM, 16 kHz, 320 frames

Every node has a different PCM digest. Each derivative stores the parent digest and transform release. The transcript and alarm remain attached to the source 8 kHz frame clock; they are not silently copied onto the resampled artifact.

First principles: identify every audio layer

Encoded artifact

The exact uploaded, fetched, or recorded bytes. Its SHA-256 digest is the durable acquisition identity. The extension and media type are declarations, not a decoder selection.

Container

RIFF/WAVE, Ogg, ISO BMFF/MP4, Matroska/WebM, AIFF, CAF, or another structure that contains tracks, packets, timing, metadata, and sometimes indexes. “WAV” is not synonymous with uncompressed PCM; WAVE can identify different audio encodings.

Microsoft’s RIFF overview defines a RIFF file as chunks with FOURCC identifiers. WAVE commonly contains fmt and data chunks, but chunk order, padding to word boundaries, additional chunks, and format variants matter. Do not locate PCM by searching for the bytes data.

Track

One independently coded stream in a container. A media file may contain several audio tracks with different languages, commentary, accessibility mixes, sample rates, or channel layouts. “Use the first audio track” is a product policy, not a universal truth.

Codec and packets

The codec turns packet bytes into decoded samples. PCM is a coding family even though it is uncompressed. FLAC is lossless. Opus, AAC, MP3, and Vorbis are lossy and stateful in different ways. Packets are codec/container units; they are not necessarily fixed-duration user-facing segments.

Decoded playback PCM

A sequence of frames under a sample rate and channel layout after the selected gapless/delay/padding policy. This is the chapter’s canonical signal for downstream transforms.

Derived PCM

A downmix, resample, denoise, separation, loudness adjustment, or crop creates new samples and needs a new identity plus provenance. “Still the same recording” does not mean byte-identical or coordinate-identical.

Observations

ASR words, diarization turns, speaker hypotheses, language labels, sound events, music tags, and embeddings are model outputs attached to ranges. They do not replace the waveform.

The safe flow is:

immutable encoded bytes
  └─ probe container + select track + decode codec
      └─ finite playback PCM under source sample clock
          ├─ ranged speech/event observations
          ├─ explicit channel transform
          ├─ explicit resample/new clock
          └─ windows/features/embeddings with exact parent ranges

Step 1: constrain the decoder surface

The companion uses:

symphonia = {
  version = "0.6",
  default-features = false,
  features = ["pcm", "wav"]
}

Symphonia’s documentation describes separate feature flags for containers and codecs. That distinction is useful security architecture: enabling WAVE does not mean enabling every codec, and enabling PCM does not mean accepting every container.

The documentation also warns that SemVer compatibility is not guaranteed. The book therefore pins the resolved graph in Cargo.lock, runs decoder fixtures during upgrades, and records:

mosaic-audio-decode-v1+symphonia-0.6+wav-pcm+gapless+interleaved-f32

The implementation gives Symphonia an empty probe hint. It does not pass a filename extension that could bias selection. After probing, it explicitly requires format_info().short_name == "wave". It selects the default audio track and resolves the registered decoder name, then requires the name to start with pcm_.

That string check is appropriate only inside this pinned narrow adapter. A broad production registry should use an explicit set of codec IDs and profiles; friendly names are not a durable policy protocol.

Why not enable every codec immediately?

Each additional path needs:

valid, truncated, corrupt, oversized, and adversarial container fixtures;
codec-specific frame and packet bounds;
delay/padding/gapless tests;
seek and pre-roll tests;
channel-layout and sample-format fixtures;
metadata conflict rules;
CPU and memory benchmarks;
license/patent review where relevant;
decoder upgrade and differential tests.

The correct goal is an intentional media capability matrix, not “anything FFmpeg can open.”

Step 2: enforce independent resource limits

AudioDecodePolicy contains:

pub struct AudioDecodePolicy {
    pub max_encoded_bytes: usize,
    pub max_sample_rate_hz: u32,
    pub max_channels: u16,
    pub max_frames: u64,
    pub max_interleaved_samples: u64,
    pub max_decoded_bytes: u64,
    pub max_packets: u64,
    pub max_observations: usize,
}

These bounds are not redundant.

Encoded bytes

Bounds the in-memory source passed to the probe. A compressed source can expand greatly, so this is not a PCM memory bound.

Sample rate

Bounds frames per second. An unexpectedly high rate amplifies duration into frame count and changes filter/inference cost.

Channels

Bounds samples per frame. A 64-channel recording has the same duration and frame count as mono but 64 times the scalar samples.

Frames

Bounds playback length under the actual decoded clock. max_frames / sample_rate is the signal duration ceiling.

Interleaved samples

The checked product is:

scalar samples = frames × channels

Calling both values “samples” is a common capacity bug. This book uses frame for one time position across every channel and scalar sample for one channel value.

Decoded bytes

The canonical representation is interleaved f32, so:

PCM bytes = frames × channels × 4

This limits the final vector, not all decoder internals.

Packets

A malformed stream can contain many tiny packets/chunks. Byte and frame caps alone do not bound loop overhead. The packet counter fails before wrapping.

Observations

Even a short signal can produce pathological token/event output. Speech and event streams have independent collection caps and per-observation text/label bounds.

The library validates policy values against absolute ceilings so a caller cannot configure usize::MAX as “unlimited.” Before each vector growth it computes the exact total and uses try_reserve.

What these bounds do not cover

The probe, demuxer, and decoder allocate internal buffers before the application sees each decoded buffer. CPU can also be pathological. For hostile production input, run the media worker with:

admission/concurrency limits;
process or container memory and CPU quotas;
wall-clock cancellation enforced outside the worker;
no network and minimal filesystem access;
patched dependency graph and fuzzed adapters;
per-format/codec failure telemetry;
bounded logs that never include private waveform content by default.

Memory-safe Rust reduces one class of risk; it does not make untrusted parsing free.

Step 3: trust decoded buffers, not declarations

Container/track codec parameters may declare a sample rate, channel layout, bit depth, and frame count. The boundary uses them for early rejection when available, but establishes the canonical specification from the first decoded audio buffer.

Every later decoded buffer must have the same AudioSpec. A midstream rate or channel change returns InvalidOrChangingSpec; the simple artifact cannot represent it.

This rule matters because:

declarations can be missing or wrong;
codecs can signal behavior during decoding;
chained streams can change configuration;
a decoder can return a more accurate actual output rate/layout.

If changing specifications are a product requirement, represent segments:

segment 0: source packet/time range → PCM spec A
segment 1: discontinuity/change receipt
segment 2: source packet/time range → PCM spec B

Do not append different rates into one vector and attach one sample clock.

Decode errors are not silence

Symphonia’s general player example can skip recoverable packet decode errors. An evidence pipeline cannot do that silently. This boundary fails the artifact on an I/O or decode error.

Alternative policies are valid only when explicit:

reject the complete artifact;
preserve a gap with exact duration and reason;
apply codec packet-loss concealment and record it;
salvage a prefix and mark it incomplete;
route to human/manual recovery.

Dropping a damaged packet and concatenating later PCM shifts every following frame and corrupts citations.

Step 4: make codec delay and gapless semantics visible

Encoders and transform codecs can produce decoder output that should not be played at the beginning or end. Track metadata may include delay and padding. Seeking can require decoding earlier packets to reconstruct state.

The companion constructs Symphonia’s decoder with gapless(true). Its AudioDecoderOptions defines gapless mode as trimming delay or padding frames. PcmLineage::Decoded records gapless: true.

For this chapter’s PCM WAVE fixture, no codec delay exists. The contract becomes essential when new codecs are enabled.

Ogg Opus is a precise example. RFC 7845 defines granule positions on a 48 kHz clock, a pre-skip count that is decoded then discarded, PCM sample position as granule position minus pre-skip, final-page end trimming, and seek pre-roll. Therefore:

first decoded buffer offset is not automatically playback time zero;
packet timestamp is not automatically the first returned PCM frame;
seeking directly to the target packet is insufficient;
container granule, decoder output, and playable PCM need separate coordinates.

FLAC has different framing and verification behavior; the current standard is RFC 9639. A FLAC adapter should enable verification where supported, preserve stream/frame metadata, and test the streamable-subset constraints.

Step 5: preserve channel meaning

Symphonia returns a channel set, not only a count. The example records:

Positioned(Position(FRONT_LEFT | FRONT_RIGHT))

The debug representation is diagnostic. A durable schema should store a versioned list or standard channel mask, not library-formatted text.

Channel meaning affects:

spatial evidence and speaker position;
accessibility mixes;
center-channel dialogue;
low-frequency-effects handling;
beamforming/microphone arrays;
phase and polarity;
ASR model input;
loudness weighting.

The Web Audio API channel rules illustrate why channel ordering and downmix matrices must be specified. Mono, stereo, quad, and 5.1 have defined speaker layouts; arbitrary channel arrays need explicit interpretation.

The companion’s downmix is intentionally simple

equal_weight_mono() computes:

mono[frame] = sum(channel samples at frame) / channel_count

It produces a new mono PCM artifact with:

unchanged sample rate and frame count;
layout MONO_EQUAL_WEIGHT;
a new PCM digest;
parent PCM digest;
input channel count;
release mosaic-audio-mix-v1+equal-weight-average.

This is deterministic and bounded. It is not a universal perceptual downmix:

opposite-polarity channels can cancel;
LFE should not necessarily have equal gain;
dialogue center may need a different coefficient;
microphone arrays should often be beamformed, not averaged;
accessibility/descriptive tracks should remain selectable;
peak headroom and loudness need evaluation.

For a supported speaker layout, implement a named matrix with explicit coefficients, accumulation precision, clipping/headroom behavior, and golden signals. For unknown/unpositioned layouts, reject or require a product choice.

Step 6: make PCM identity self-describing

Hashing only f32 bytes is ambiguous. The same bytes under 8 kHz mono and 16 kHz mono represent different durations. The companion’s PCM digest uses a domain-separated binary stream containing:

"mosaic-interleaved-f32-pcm-v1\0"
sample_rate_hz as little-endian u32
channel_count as little-endian u16
channel-layout byte length + bytes
scalar-sample count
every finite f32 bit pattern in interleaved frame-major order

It hashes f32::to_bits() in little-endian order. It rejects NaN and infinities.

This derived digest is not the encoded artifact digest. Keep both:

Digest	Identifies
Encoded SHA-256	exact acquired RIFF/WAVE bytes including metadata and coding
Playback PCM SHA-256	exact finite signal under rate/layout/sample schema
Mixed PCM SHA-256	exact downmix output
Resampled PCM SHA-256	exact output under the resampling release

A provenance receipt links them. PCM hash equality does not establish who recorded the audio, consent, authenticity, capture time, or semantic equivalence.

Step 7: use sample frames as authoritative time

The sample clock is exact:

time_seconds = frame_boundary / sample_rate_hz

But decimal microseconds often cannot represent that fraction exactly. At 44.1 kHz one frame is approximately 22.6757 µs. Repeated floating addition accumulates error and makes boundary behavior unclear.

The companion defines:

pub struct SampleRange {
    pub start_frame: u64,
    pub end_exclusive_frame: u64,
}

pub struct TimeRangeMicros {
    pub start_us: u64,
    pub end_exclusive_us: u64,
}

SampleRange is authoritative for the source PCM. It is nonempty and must fit within decoded frames.

Sample range to covering time

For [start,end):

start_us = floor(start × 1,000,000 / rate)
end_us   = ceil(end   × 1,000,000 / rate)

The resulting microsecond range covers every selected sample interval. It may be slightly wider because of rounding.

Time range to covering samples

For a valid microsecond query:

start_frame = floor(start_us × rate / 1,000,000)
end_frame   = ceil(end_us   × rate / 1,000,000)

The result is clamped at the exact signal boundary and must be nonempty. Calculations use u128 intermediates and checked conversions.

This is a coverage mapping, not a mathematical inverse. Rounding means:

samples → microseconds → samples

can include an additional boundary sample at rates that do not divide one million. Store source sample coordinates with every observation and derive friendly time for display.

Wall time is another coordinate system

If a capture began at an acquired wall-clock instant, sample frame zero can map to that clock only under a capture-clock model. Hardware clocks drift; buffers can be dropped; device timestamps can jump; network streams have jitter. Record:

capture clock identity and frequency;
monotonic device timestamps;
discontinuities/dropped frames;
clock-offset/drift estimates and uncertainty;
wall-clock synchronization source;
transformations into the playback PCM clock.

Do not add acquired_at + frame/rate and call it exact without that evidence.

Step 8: resample as a versioned signal-processing transform

Resampling changes the number and values of frames. Downsampling also requires a low-pass filter to avoid aliasing frequencies above the target Nyquist limit.

The companion pins:

rubato = "4"

and uses Fft<f32> with:

fixed input/output ratio;
FixedSync::Both;
1,024-frame chunk-size hint;
Rubato’s default Blackman-Harris-2 anti-aliasing window;
process_all, which resets the resampler, processes a complete in-memory clip, and trims startup delay and trailing padding.

Rubato’s documentation recommends the FFT resampler for offline fixed-rate conversion and warns that one oversized process() call does not trim startup delay. That is why the companion calls process_all.

Before constructing the resampler, it computes:

expected output frame ceiling =
  ceil(input_frames × output_rate / input_rate)

It rejects a zero/same/excessive target, checks the output-frame budget, computes channel-scaled f32 bytes, and checks memory. After processing, it validates nonempty interleaved shape, finite samples, actual output frames, and output digest.

The release is:

mosaic-audio-resample-v1+rubato-4+fft+blackman-harris2+fixed-both

The result stores parent PCM digest plus input/output rates.

Fixed rate versus drifting clocks

An offline 44.1→48 kHz conversion has one fixed rational ratio. Two live devices nominally at 48 kHz have independent oscillators; their ratio changes slightly over time. Rubato distinguishes:

synchronous FFT for a fixed ratio;
asynchronous sinc for a changing high-quality ratio;
asynchronous polynomial when CPU matters and aliasing/distortion trade-offs are accepted;
Slip for very small clock differences, which occasionally inserts/drops a frame and is not a true resampler.

A live ingest path needs a drift estimator, bounded ratio updates, buffer-level controller, discontinuity policy, latency target, and realtime-safe preallocated processing. Reusing the offline method for a never-ending stream would accumulate latency or underflow/overflow buffers.

Resampling and timestamps

A source observation should keep its source SampleRange. To map it to resampled PCM, use the transform’s exact boundary mapping and its filter-delay/trim contract, then record the mapped derived range. Do not copy frame numbers between sample rates:

source frame 80 at 8 kHz  = 10 ms
derived frame 80 at 16 kHz = 5 ms

The example intentionally does not auto-copy observations onto the derived signals.

Step 9: attach speech as evidence-backed observation

SpeechObservation contains:

pub struct SpeechObservation {
    pub text: String,
    pub samples: SampleRange,
    pub confidence_basis_points: u16,
    pub method_release: String,
    pub language: Option<LanguageLabel>,
    pub speaker_label: Option<String>,
    pub ordinal: u32,
}

The boundary rejects empty/NUL/oversized text, invalid source ranges, confidence above 10,000 basis points, malformed method/speaker labels, duplicate ordinals, and collection overflow.

These fields still need precise semantics:

Does the range cover an utterance, segment, word, or token?
Is confidence sequence-level, average token posterior, calibrated correctness, or vendor score?
Is the speaker label a diarization-local cluster or authenticated person identity?
Is language observed per segment or forced by request?
Was punctuation generated, restored, or present in a reference?
Were timestamps emitted by alignment, attention heuristics, CTC, or forced alignment?

speaker_label: "operator-1" is not proof of a real person. Diarization answers “same/different speaker cluster under this method”; speaker verification/identification is a different biometric, consent, security, and evaluation problem.

ASR text then enters the text-normalization chapter as a derived text artifact. Preserve the raw model response digest and token/word alignment if available. Human correction creates a new assertion linked to the original speech observation; do not mutate history.

Evaluate speech beyond one WER number

Word error rate is based on substitutions, deletions, and insertions relative to a normalized reference:

WER = (S + D + I) / N

But reference normalization can dominate results across punctuation, casing, numbers, compounds, and scripts. Report the exact scorer/normalizer and include:

character error rate where word segmentation is unstable;
timestamp boundary error and coverage;
named-entity/number/domain-term accuracy;
language, accent/dialect, device, noise, overlap, duration, and code-switch slices;
hallucinated speech in silence/music;
confidence calibration and abstention;
diarization error under a stated collar/overlap policy;
downstream task success, latency, and cost.

NIST’s speech evaluations exist because common data and scoring protocols are essential. A product needs its own rights-cleared, task-representative held-out audio and production-failure slices.

Step 10: attach non-speech events independently

AudioEventObservation contains label, source sample range, basis-point confidence, method release, and ordinal. The executable labels frames [80,160) as an alarm.

Speech and non-speech events use separate streams because:

their taxonomies and calibration differ;
events can overlap speech;
one event can contain many utterances;
an ASR “no speech” result does not prove silence;
music, impact, siren, glass break, laughter, and machinery have different window needs.

Event models often consume overlapping windows. Preserve:

analysis window and hop in source frames;
resampling/mix/filter preprocessing;
raw per-class scores before thresholding when policy allows;
taxonomy and model release;
threshold/calibration release;
merge/hysteresis policy that turns windows into intervals;
negative/unknown/abstained outcomes.

Avoid turning the highest softmax class into truth when all classes are poor matches.

RMS is not loudness

The companion includes rms_by_channel(range) to prove correct frame/channel traversal:

RMS(channel) = sqrt(sum(sample²) / selected_frames)

RMS is useful for fixtures and signal energy. It is not perceived loudness, true peak, clipping risk between samples, or speech intelligibility.

EBU R 128 version 5 defines a loudness-normalization ecosystem around ITU-R BS.1770 measurement, including programme loudness, loudness range, and maximum true peak. EBU Mode includes momentary, short-term, and integrated measurements. If the product needs loudness normalization:

use a validated standard implementation and official test signals;
record standard/release, gating, channel weighting, target, gain, limiter, and true-peak behavior;
preserve the pre-normalized artifact;
evaluate ASR/event performance before and after;
never call raw RMS “LUFS.”

Failure investigations

“Duration says 10 seconds, but the decoded signal has 9.98”

Compare container duration, track timebase/duration, playable frame count, sample rate, decoder delay, end padding, gapless setting, corruption, and salvage policy. Decide which coordinate the UI and model use. Do not stretch samples just to match metadata.

“Transcript timestamps drift later in the file”

Check sample rate used by the recognizer, input frame count, resampling output length, capture-clock drift, dropped buffers, chunk overlap, timestamp rounding, and whether each segment resets a local clock. A constant offset suggests delay/pre-skip; increasing error suggests a clock-rate mismatch.

“Stereo speech disappeared after mono conversion”

Inspect channel layout and phase. Equal-weight averaging can cancel opposite-polarity signals. Replay each channel independently, inspect correlation, and use a layout-/microphone-aware transform.

“Downsampled audio contains tones that were not present”

Suspect aliasing or a polynomial/linear resampler without sufficient anti-alias filtering. Inspect input frequency content, target Nyquist frequency, resampler release/settings, and spectrogram fixtures.

“The transcript skips a section but later timestamps look continuous”

Check packet decode errors and whether the adapter discarded a packet. The companion rejects rather than splicing. If concealment or gaps are allowed, the timeline must preserve them.

“Two PCM buffers sound the same but have different hashes”

Check rate, layout label, sample count, f32 bit patterns, decoder/resampler release, dithering, gain, channel order, signed zero, and transform history. Perceptual equivalence is a different comparison/eval.

“The ASR confidence is 0.98 but the transcript is wrong”

Determine what confidence means and whether it is calibrated on this domain/language/noise slice. Vendor/model scores are not automatically probabilities. Use held-out reliability curves and abstention/escalation thresholds.

Audio can contain voices, background conversations, locations, devices, health information, and biometric characteristics. A production boundary needs:

collection and retention purpose;
participant/voice consent where required;
tenant-scoped access and encrypted storage;
redaction policy for transcripts and raw recordings;
audit trails for playback/export;
strict external-provider data boundaries;
no content in ordinary logs/traces;
deletion propagated to derived PCM, transcripts, events, embeddings, and indexes;
separate authorization for speaker recognition or voice cloning.

Acoustic classifiers can also trigger consequential actions. An “alarm” observation should be input to a policy/verifier, not direct authority to call emergency services or unlock a door.

Testing strategy

The companion has twelve focused audio tests:

real in-memory RIFF/WAVE PCM probing and decoding;
invalid policy, encoded-size, and unknown-container failure;
independent sample-rate, channel, frame, scalar-sample, and decoded-byte bounds;
exact half-open frame↔microsecond coverage;
empty and out-of-bounds range rejection;
per-channel RMS over an exact range;
valid speech and event observations;
malformed and duplicate observation rejection;
explicit equal-weight mono lineage and identity;
bounded finite FFT resampling with exact 8→16 kHz duration/frame result;
invalid/same-rate/underbudget resample rejection;
PCM digest sensitivity to layout and samples.

The production fixture suite should add:

RIFF odd-byte chunk padding, unknown chunks, chunk reordering, truncation, RF64, and extensible channel masks;
every permitted PCM width/format and out-of-range float PCM;
multichannel golden layouts and downmix matrices;
FLAC integrity, seek, and streamable-subset fixtures;
Opus pre-skip, end trim, chaining, gaps, and seek pre-roll;
corrupt packets under reject/gap/concealment policies;
44.1↔48 kHz spectral, impulse, delay, and exact-length resampler tests;
clock-drift simulation for live streams;
ASR silence/music hallucination, overlapping speakers, numbers, languages, and timestamps;
acoustic events near window boundaries and overlapping speech;
EBU official loudness test signals if loudness is implemented.

Property tests should generate safe rates/frame ranges and prove containment and monotonicity of coverage mapping. Fuzz the container boundary in an independent process.

Exercises

Exercise 1: make a typed playback-PCM manifest

Replace the diagnostic channel string with a versioned channel-position enum/list. Include source artifact, decoder release, gapless policy, sample schema, rate, frames, channel order, and PCM digest in a domain-separated manifest identity.

Exercise 2: add WAVE_FORMAT_EXTENSIBLE fixtures

Decode 5.1 PCM with a channel mask. Prove correct channel order. Reject a missing/inconsistent mask instead of treating six unnamed channels as a speaker layout.

Exercise 3: add FLAC under a separate policy

Enable only Symphonia’s FLAC feature, verify when supported, add RFC 9639 fixtures, and compare decoded PCM identity against a trusted reference decoder.

Exercise 4: add Opus timing

Implement Ogg Opus in an isolated adapter/runtime if the selected decoder supports it. Preserve granule position, pre-skip, end trim, seek pre-roll, and playback-frame mapping. Test a cropped stream whose initial PCM position is nonzero.

Exercise 5: implement a speaker-layout downmix

Define one exact stereo or 5.1→mono matrix with coefficient source, accumulation precision, headroom/clipping behavior, and golden polarity/LFE/dialogue fixtures. Keep equal-weight version 1 unchanged.

Exercise 6: evaluate an ASR and event model

Use the later runtime boundary to process frozen speech, silence, music, noise, overlap, and event fixtures. Store source frame ranges and preprocessing receipts. Measure WER/CER, timestamp error, hallucinations, event precision/recall, calibration, latency, and cost.

Solution sketches

Exercise 1

Use a strict serialized schema and length-prefixed hash fields. Bind f32 layout, interleaving, endianness of the hash representation, finite-sample policy, and frame count. Do not hash incidental debug text or JSON field order.

Exercise 2

Treat count and position as different. Map the container mask into a fixed canonical position order, then verify the decoded buffer reports the same meaning. Unknown positions should remain typed unknowns or reject under the model-input policy.

Exercise 3

Add flac without all. Bound metadata/picture blocks separately from audio. Turn on decoder verification when possible; preserve verification outcome. Decode the same valid fixtures with the pinned comparison implementation and hash the canonical PCM schema.

Exercise 4

Keep container granule timestamps, decoded frames before trim, and playable PCM frames distinct. Apply pre-skip exactly once. Seeking starts early enough for pre-roll, discards to the requested playback frame, and records the actual start.

Exercise 5

A matrix is data plus policy, not nested ifs. Validate required positions, multiply each frame in f64 or specified precision, apply explicit gain/headroom, then convert. Test impulses in one channel at a time and correlated/anticorrelated signals.

Exercise 6

Freeze reference normalization before scoring. Treat word timestamps and utterance timestamps as different outputs. Slice by signal condition. Calibrate thresholds on development data, choose them against product costs, and report untouched held-out results.

Completion checklist

References

Symphonia 0.6 documentation — containers, codecs, feature flags, and compatibility warnings
Symphonia AudioDecoderOptions — gapless and verification options
Rubato 4 documentation — fixed/asynchronous resampler selection, process_all, anti-aliasing, delay, and realtime allocation guidance
Microsoft RIFF overview — chunk structure, FOURCCs, fmt , and data
RFC 7845: Ogg Encapsulation for Opus — granule positions, pre-skip, PCM positions, end trim, and seek pre-roll
RFC 9639: FLAC — current lossless codec/format standard
Web Audio API 1.1 — sample rates, channel positions, and mixing model
EBU R 128 loudness resources and version 5 recommendation
Companion implementation: crates/mosaic-evidence/src/audio.rs and crates/mosaic-evidence/examples/audio_pipeline.rs

What comes next

Video combines a container demuxer, one or more video/audio tracks, independent time bases, decode-versus-presentation order, variable frame duration, color/geometry, cuts, motion, and sampling. The next chapter builds a checked video timeline before extracting “one frame every N seconds,” because that shortcut is not a reliable representation of variable-frame-rate evidence.

Audio — Codecs, Resampling, Channels, Timestamps, Speech, and Events

Learning objectives

Prerequisites and dependency order

Completion artifact

First principles: identify every audio layer

Encoded artifact

Container

Track

Codec and packets

Decoded playback PCM

Derived PCM

Observations

Step 1: constrain the decoder surface

Why not enable every codec immediately?

Step 2: enforce independent resource limits

Encoded bytes

Sample rate

Channels

Frames

Interleaved samples

Decoded bytes

Packets

Observations

What these bounds do not cover

Step 3: trust decoded buffers, not declarations

Decode errors are not silence

Step 4: make codec delay and gapless semantics visible

Step 5: preserve channel meaning

The companion’s downmix is intentionally simple

Step 6: make PCM identity self-describing

Step 7: use sample frames as authoritative time

Sample range to covering time

Time range to covering samples

Wall time is another coordinate system

Step 8: resample as a versioned signal-processing transform

Fixed rate versus drifting clocks

Resampling and timestamps

Step 9: attach speech as evidence-backed observation

Evaluate speech beyond one WER number

Step 10: attach non-speech events independently

RMS is not loudness

Failure investigations

“Duration says 10 seconds, but the decoded signal has 9.98”

“Transcript timestamps drift later in the file”

“Stereo speech disappeared after mono conversion”

“Downsampled audio contains tones that were not present”

“The transcript skips a section but later timestamps look continuous”

“Two PCM buffers sound the same but have different hashes”

“The ASR confidence is 0.98 but the transcript is wrong”

Security, privacy, and consent

Testing strategy

Exercises

Exercise 1: make a typed playback-PCM manifest

Exercise 2: add WAVE_FORMAT_EXTENSIBLE fixtures

Exercise 3: add FLAC under a separate policy

Exercise 4: add Opus timing

Exercise 5: implement a speaker-layout downmix

Exercise 6: evaluate an ASR and event model

Solution sketches

Exercise 1

Exercise 2

Exercise 3

Exercise 4

Exercise 5

Exercise 6

Completion checklist

References

What comes next