Speech Generation, Voice Consent, and Audio Evaluation

Speech generation turns language into a time-bearing artifact. That makes it more than “text with a voice.” The system must decide what the text means, how it is pronounced, where words begin and end, how long every sound lasts, how pitch and energy move, whose vocal identity is represented, how waveform samples are produced, and whether publishing that voice is authorized.

This chapter implements those boundaries in Rust. The companion normalizes a deliberately narrow English fixture language, constructs sample-exact voiced/unvoiced/silence units, varies prosody from a named seed, synthesizes a bounded waveform, encodes a real 16-bit PCM/WAV file, validates its header and length, and emits a content-addressed receipt. A second, non-public test path models a consented replica voice and fails closed on missing, mismatched, expired, purpose-incompatible, or revoked authorization.

The waveform is a source-filter teaching signal, not a trained TTS model and not human-like speech. It exists to make timing, amplitude, encoding, identity, and authorization inspectable. A production acoustic model and neural vocoder can replace the toy signal generator without removing those contracts.

Learning objectives

By the end you will be able to:

decompose TTS into document parsing, text normalization, linguistic analysis, pronunciation, acoustic planning, vocoding, encoding, validation, and publication;
explain why text normalization and grapheme-to-phoneme conversion are language-specific;
compare autoregressive, non-autoregressive, end-to-end, codec-token, and streaming speech models;
connect duration, fundamental frequency, energy, mel spectrograms, phase, and waveform samples;
implement sample-exact segment timing without accumulated floating-point clock drift;
distinguish voice identity, speaking style, language, accent, emotion, and recording conditions;
model voice consent as revocable, purpose-bound authorization rather than metadata;
understand why a speaker-similarity score cannot grant consent;
design safe SSML and lexicon boundaries;
publish a validated PCM/WAV artifact with exact format and digest;
measure intelligibility, naturalness, pronunciation, prosody, speaker similarity, artifacts, latency, and safety separately;
design streaming synthesis without clicks, missing audio, or dishonest completion;
preserve generated-audio lineage and Content Credentials without claiming factual authenticity;
investigate repetition, skips, mispronunciation, clipping, silence, identity leakage, and drift.

Prerequisites and dependency order

Complete audio evidence processing first. It establishes byte-sniffed decoding, PCM layout, channels, sample clocks, resampling, and ranged observations. Complete provenance before publishing generated artifacts, and sampling and reproducibility before attaching guarantees to a prosody seed.

Image generation introduced bundle identity and derived-media provenance. Speech adds an authorization dimension: a technically valid model may represent a voice the caller has no right to synthesize for the requested purpose.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example speech_generation

The command writes:

target/generated-audio/<wav-sha256>.wav

The pinned fixture produces:

text frontend     NFC + bounded fixture English ASCII
voice origin      designed_synthetic
sample rate       16,000 Hz
channels          1
sample format     signed PCM, 16 bits
frames            40,800
duration          2,550,000 µs
prosody seed      42
peak              0.542489 full scale
RMS               0.250300 full scale
clipped samples   0
WAV bytes         81,644
WAV SHA-256       5c55db828b25ee18f83a1e20c01976af6dcd363636c548168240fb2777ef7b7f

Run the focused tests:

cargo test -p mosaic-ml --all-features speech_generation
cargo test -p mosaic-ml --all-features --example speech_generation

The implementation is crates/mosaic-ml/src/speech_generation.rs; the executable is crates/mosaic-ml/examples/speech_generation.rs.

A speech system is a sequence of decisions

The SSML 1.1 Recommendation describes a useful conceptual pipeline: parse the document, analyze structure, normalize written forms, derive pronunciations, build prosody, and produce a waveform. A modern neural system may train several stages jointly, but the decisions still exist:

Input interpretation decides whether content is plain text or trusted markup.
Segmentation finds paragraphs, sentences, tokens, languages, and speaker turns.
Normalization expands dates, numbers, currencies, units, abbreviations, URLs, and symbols.
Pronunciation maps words or characters into language-specific phonemes or learned units.
Linguistic encoding represents context, stress, syntax, semantics, and style instructions.
Alignment/duration decides how many acoustic frames belong to each unit.
Acoustic generation predicts mel frames, continuous latents, or audio tokens.
Vocoder/decoder creates time-domain samples.
Postprocessing applies gain, loudness policy, trimming, and perhaps watermarking.
Encoding creates WAV, Opus, AAC, FLAC, or another declared artifact.
Validation independently checks decodability, timing, amplitude, and policy.
Publication stores the bytes, receipt, authorization decision, and provenance.

Collapsing those stages behind one string response makes failures hard to localize. A skipped word could be normalization, pronunciation, alignment, acoustic-model attention, vocoder corruption, stream assembly, or output truncation.

Text is not pronunciation

Normalize meaning, not characters

Consider:

Dr. Lee paid $1,250 on 03/04.

Should Dr. mean “doctor” or “drive”? Is the amount US dollars? Is the date March fourth or April third? Normalization depends on language, locale, domain, surrounding structure, and sometimes external metadata. Lowercasing and deleting punctuation is not a text frontend.

A production normalizer should produce an auditable intermediate representation:

source span → normalized token → interpretation → rule/model release → confidence

For high-consequence content such as medication doses, financial amounts, emergency directions, or legal notices, ambiguous expansion should require explicit author intent or review. Retain a map to source bytes so a pronunciation failure can be shown in its original context.

The companion intentionally accepts only ASCII letters, spaces, and a small punctuation set. It rejects digits and non-ASCII language rather than pretending to pronounce them. It applies NFC, collapses ASCII whitespace, lowercases, and hashes the normalized result. That is an honest fixture contract, not a general English frontend.

Graphemes, phonemes, and contextual pronunciation

A grapheme is a writing-system unit; a phoneme distinguishes sounds in a language. Their mapping is not one-to-one. English “read” changes pronunciation with tense. A letter can be silent. One sound can use several letters. Mandarin tone changes lexical meaning. Arabic and Hebrew writing may omit vowels. Japanese mixes writing systems. Code-switching can change pronunciation inside one sentence.

A pronunciation boundary needs:

language and locale per span;
text-normalizer release;
grapheme-to-phoneme model and lexicon releases;
phoneme alphabet;
fallback policy for unknown words;
user pronunciation overrides;
stress, syllable, and tone representation;
explicit failure when the requested language is unsupported.

Do not infer “supported language” from the fact that a tokenizer emitted IDs. Evaluate native speakers across dialects, names, loanwords, numbers, and code-switching.

Treat SSML as active input

SSML can express language, voice, pronunciation, substitutions, breaks, emphasis, prosody, and embedded audio. It is also XML and can reference resources. Never concatenate untrusted text into trusted SSML.

For a public API:

separate PlainText and TrustedSsml request types;
use a hardened parser with entity expansion and external access disabled;
bound document depth, node count, attribute size, phoneme count, break duration, and total predicted speech;
allowlist elements and attributes supported by the selected release;
restrict or prohibit remote audio and lexicon fetching;
resolve resources through an authorized content-addressed store;
include the SSML profile/version and normalized document digest in the receipt.

Escaping protects XML syntax; it does not authorize a caller to select another person’s voice or request a 24-hour pause.

Acoustic representations

Samples and time

At sample rate (f_s), frame index (n) represents:

[ t_n = \frac{n}{f_s} ]

A 2.55-second mono signal at 16 kHz has:

[ 2.55 \times 16{,}000 = 40{,}800 \text{ frames} ]

Use integer frames as the primary clock. Convert to microseconds with checked rational arithmetic:

[ \text{duration}_\mu = \left\lfloor\frac{\text{frames}\times1{,}000{,}000}{f_s}\right\rfloor ]

Repeatedly adding floating-point seconds can create boundary gaps or overlaps. The companion makes each unit’s start_frame equal the preceding end_frame and tests that the final endpoint equals the WAV frame count.

Spectrograms

Many TTS systems predict a mel spectrogram rather than samples. A short-time Fourier transform windows overlapping waveform segments and produces complex frequency bins. Magnitude is mapped through a mel-spaced filter bank and often log-compressed. Typical model tensors look like:

[batch, mel_bins, acoustic_frames]

The representation omits or deemphasizes phase, so a vocoder must infer a plausible waveform. Release identity includes sample rate, FFT size, window, hop, mel bins, minimum/maximum frequency, log convention, normalization, and tensor layout. “80-bin mel” is not enough.

If hop length is 256 samples at 24 kHz, one acoustic frame spans about 10.67 ms. Duration prediction maps input units to integer counts of those frames. A one-frame error repeated across many units can change rhythm materially.

Duration, pitch, and energy

Speech is one-to-many: the same sentence can be spoken with different timing and expression. FastSpeech 2 explicitly conditions on duration, pitch, and energy to model part of that variation.

Duration controls alignment and speaking rate.
Fundamental frequency (F_0) approximates the voiced excitation rate and carries intonation.
Energy affects prominence and perceived emphasis.
Pauses communicate structure and turn-taking.

Pitch shifting is logarithmic. A shift of (c) cents maps frequency (f) to:

[ f’ = f \times 2^{c/1200} ]

An octave is 1,200 cents. Changing pitch should not silently change duration. The companion tests those controls independently.

Acoustic model and vocoder architectures

Two-stage autoregressive systems

Tacotron 2 uses a recurrent sequence-to-sequence model to predict mel spectrograms and a WaveNet-family vocoder to synthesize waveform samples. Autoregressive attention can sound natural, but inference is sequential and alignment may repeat, skip, or fail on long or unusual text. WaveNet models waveform samples autoregressively with dilated causal convolutions; direct sample-by-sample generation is expensive.

Non-autoregressive systems

Duration-informed models expand text representations to acoustic-frame length and predict many frames in parallel. They offer predictable alignment and lower latency, but duration errors become explicit. A separate vocoder still turns acoustic features into samples.

HiFi-GAN is an adversarial neural vocoder designed for efficient, high-fidelity waveform synthesis from mel spectrograms. A vocoder cannot repair omitted words or a wrong pronunciation from the acoustic plan. Evaluate stages separately.

End-to-end latent systems

VITS combines variational inference, normalizing flows, adversarial training, and a stochastic duration predictor in an end-to-end architecture. Joint training can reduce mismatch between acoustic model and vocoder, but production still needs observable input, timing, speaker, sampling, output, and authorization contracts.

Codec-token and language-model systems

Newer systems quantize audio into discrete codec tokens and model them with autoregressive or parallel sequence architectures. They can support long-form generation, zero-shot voices, and expressive continuation, while adding codec release, codebook, token-rate, multi-stream ordering, prompt-audio, and sampling contracts. Token validity does not guarantee decodable or safe audio.

Select architecture based on evaluated requirements:

Requirement	Likely pressure
low first-audio latency	streaming or parallel acoustic generation
exact word timing	explicit duration/alignment model
long-form stability	chunk planning and cross-chunk state
expressive diversity	stochastic latent/prosody model
edge deployment	compact model, efficient vocoder, lower sample rate
zero-shot voice	reference encoder plus substantially stronger consent controls
deterministic fixtures	explicit frontend, duration, RNG, and sample pipeline

Walk through the Rust companion

1. Release identity

SpeechModelRelease binds:

pub struct SpeechModelRelease {
    pub model_id: String,
    pub upstream_revision: String,
    pub text_frontend_sha256: String,
    pub acoustic_model_sha256: String,
    pub vocoder_sha256: String,
    pub voice_profile_sha256: String,
    pub language: String,
    pub sample_rate_hz: u32,
    pub voice_origin: VoiceOrigin,
}

A production bundle also includes tokenizer/phonemizer, lexicons, speaker encoder, codec, acoustic configuration, vocoder configuration, style adapters, precision, runtime, watermarker, output encoder, and safety models. The voice profile digest is distinct from the general model: one multispeaker model can host many independently authorized identities.

2. Preflight before planning duration

The generator rejects unsupported sample rate, speaking-rate bounds, pitch range, zero authorization day, oversized input, unsupported characters, too many units, excessive predicted frames, and excessive WAV bytes.

Duration is data-dependent. A byte limit alone does not bound audio duration: punctuation, phoneme expansion, slow speaking rate, or SSML breaks can expand short input. A production preflight should run normalization and duration estimation under an admission budget before reserving accelerator and output-storage capacity.

3. Build a sample-exact plan

Each normalized character becomes a pedagogical voiced, unvoiced, or silence unit. Base duration is scaled by speaking rate. A seeded ±1.5% pitch variation demonstrates prosody sampling. The receipt stores source byte range, unit kind, half-open frame range, and fundamental frequency in millihertz.

This is not a linguistic phoneme plan. It intentionally makes no intelligibility claim. In a real system the equivalent receipt can store word/phoneme timing, but consider privacy: detailed text timelines may reconstruct the input. Keep full alignments in access-controlled artifact storage and emit only the observability fields necessary for ordinary operations.

4. Synthesize a bounded waveform

Voiced units use a fundamental plus second and third harmonics; unvoiced units use deterministic noise; silence emits zero. A short attack/release envelope avoids discontinuities at unit edges. Every sample is checked for finiteness. Peak, RMS, and pre-clamp clipping count are computed before encoding.

A production neural vocoder follows the same outer rule:

validated acoustic tensor
    → bounded vocoder execution
    → finite waveform tensor
    → channel/sample-rate/range validation
    → gain/loudness policy
    → checked encoder

Never hide NaN with a clamp. Non-finite generation is a failed artifact and may indicate an unsupported precision, corrupt weights, unstable input, or runtime bug.

5. Encode and revalidate WAV

The companion writes a canonical 44-byte RIFF/WAVE header:

RIFF chunk
  WAVE form
  fmt  chunk: PCM, mono, 16 kHz, 16 bit
  data chunk: little-endian signed samples

The file is then parsed independently enough to verify markers, channels, sample rate, bits, declared data bytes, RIFF length, actual byte length, and frame count. The exact encoded bytes are hashed.

WAV is a container family, not one sample format. Production readers must handle chunks rather than assuming all valid files use exactly 44 header bytes. The generator can deliberately emit a narrow canonical subset while the ingestion boundary remains defensive.

Separate identity from style

These are different controls:

voice identity: who the sound resembles;
language/accent: linguistic and regional pronunciation;
style/emotion: calm, excited, whispered, formal;
prosody: timing, pitch, energy, pauses;
recording condition: microphone, room, codec, noise.

A style prompt such as “warm documentary narration” need not identify a person. A request naming or referencing a person does. Similarity can emerge unintentionally from training data or speaker interpolation, so designed voices also need identity-leakage evaluation.

Model a grant, not a checkbox

The companion’s replica path requires:

pub struct VoiceAuthorization {
    pub voice_profile_sha256: String,
    pub consent_artifact_sha256: String,
    pub permitted_purposes: Vec<SpeechPurpose>,
    pub valid_from_day: u32,
    pub valid_through_day: u32,
    pub revocation_list_sha256: String,
    pub revoked: bool,
}

Construction fails if the grant does not bind the exact voice profile. Generation fails if purpose, day, or revocation state is not authorized. The designed-synthetic fixture rejects an unrelated consent record so provenance cannot falsely imply that a human licensed it.

A production authorization service should additionally model:

verified grantor and authority to grant;
subject and voice-profile enrollment evidence;
allowed product, audience, territory, language, and content category;
commercial/advertising/political restrictions;
edit, derivative, transfer, and model-improvement permissions;
compensation or usage accounting where applicable;
minor/deceased-person safeguards;
effective time and revocation;
deletion/retention obligations;
incident and takedown contact;
immutable decision receipt and policy release.

Do not put a mutable consent=true in model metadata. Check current authorization at generation and again at publication for workflows where revocation can occur while a job is queued.

The U.S. Copyright Office’s 2024 Digital Replicas Report documents both beneficial authorized uses and serious harms from unauthorized voice replicas. It is a policy report, not a complete statement of current law in every jurisdiction. Obtain legal review for the product, location, and date of deployment.

Similarity is not permission

A speaker encoder can estimate whether two recordings sound alike. It cannot determine whether the subject consented, whether the grantor was authorized, whether the purpose is allowed, or whether consent was revoked. Conversely, a low score does not prove a replica is harmless to a listener who recognizes it.

Use speaker similarity for evaluation and leakage detection. Use signed, independently managed authorization records for permission.

Streaming without corrupting the artifact

Long-form TTS should not require the caller to wait for the complete waveform, but arbitrary chunking creates audible seams and lifecycle ambiguity.

A streaming design needs:

document/sentence planning before or during generation;
maximum lookahead and text buffering;
stable sample-rate/channel/format headers;
sequence numbers and sample ranges;
overlap or model state across acoustic chunks;
vocoder receptive-field padding;
crossfade or overlap-add policy;
backpressure;
cancellation and disconnect behavior;
final sample count and artifact digest;
an explicit terminal event.

Do not synthesize each network packet as an independent sentence fragment. Coarticulation and prosody cross word boundaries. Chunk at linguistic boundaries when possible, preserve model state, and include enough overlap to eliminate decoder transients.

Measure:

request accepted
→ text frontend complete
→ first acoustic chunk
→ first playable audio byte
→ last audio byte
→ validation complete
→ artifact published

Time to first byte is not necessarily time to first playable audio. A client may need a container header, codec preroll, or buffered frames.

On cancellation, stop future model work, join owned tasks, mark the artifact incomplete, and never publish a partial stream under a completed receipt. If partial playback is a supported product, give it a distinct partial-artifact identity and duration.

Output formats and loudness

PCM/WAV is simple and lossless but large. FLAC is lossless and compressed. Opus is efficient for speech and streaming but introduces codec delay and lossy changes. AAC/MP3 may be necessary for platform compatibility. Format selection affects:

media type and container;
sample format, rate, and channels;
encoder release and options;
delay, preroll, and final padding;
byte size and bandwidth;
waveform digest;
watermark durability;
downstream ASR/safety evaluation.

Keep a lossless archival artifact when exact waveform analysis or future transcoding matters. Treat every transcode as a derived artifact with its own digest and timing map.

Peak amplitude is not perceived loudness. Production publication may use an explicit loudness target and true-peak ceiling, measured with an approved implementation. Keep raw model output and normalized publication output distinct in lineage. Aggressive normalization can amplify noise or change watermark behavior.

Evaluation: one score is not speech quality

Intelligibility and content

Transcribe generated speech with one or more held-out ASR systems and compute word or character error rate against the intended spoken form. This catches skips, repeats, and gross pronunciation failures, but the ASR has its own language, accent, noise, and demographic biases. Evaluate exact normalized text, not the unexpanded source string.

Add targeted checks for:

names and domain terms;
numbers, dates, currencies, units, and abbreviations;
homographs and pronunciation overrides;
code-switching;
long sentences and repeated phrases;
punctuation and question intonation;
unsafe or prohibited text transformations.

Naturalness and listener judgment

Mean Opinion Score protocols ask listeners to rate quality. The ITU-T P.800 recommendation is a foundational reference for subjective speech-quality methods. A reliable study needs clear prompts, calibrated playback, randomized and blinded presentation, appropriate anchors/references, native-language listeners, sufficient ratings, and confidence intervals.

Separate questions:

Is the content intelligible?
Is pronunciation correct?
Is the speech natural?
Is prosody appropriate?
Is the voice consistent?
Does it match an authorized reference speaker?
Are artifacts annoying?
Is the style appropriate for the use?

One overall MOS hides which dimension changed.

Speaker similarity and leakage

For an authorized replica, compare generated speech to held-out enrollment/reference recordings with blinded human judgments and multiple speaker-verification models. Include negative speakers with similar age, accent, and pitch. Test unseen text, emotions, languages, and recording conditions.

For a designed voice, reverse the goal: ensure it does not cross a similarity threshold for protected reference identities. Search a curated risk set under access controls. Thresholds require calibration; embedding cosine similarity is not a universal probability.

Signal and artifact metrics

Track:

decoded sample rate, channels, format, and duration;
NaN/Inf and malformed-artifact rate;
clipping and true peak;
silence ratio and unexpected long pauses;
RMS/loudness distribution;
DC offset;
spectral holes, aliasing, tonal noise, clicks, and discontinuities;
alignment coverage and duration outliers;
ASR WER/CER by slice;
real-time factor and first-audio latency;
peak memory and failure rate.

Reference-based spectral distances can help isolate vocoder regression when parallel ground-truth recordings exist. They do not replace listening.

Spoofing and detector tests

Synthetic speech can attack speaker-verification and social trust. Test publication artifacts, telephone-bandwidth transcodes, re-recordings, background noise, compression, and adversarial postprocessing. ASVspoof provides research challenge protocols and datasets for speech deepfake detection and spoofing-robust verification.

Detection is defense in depth, not authorization or provenance. Detectors drift as generators change and can be evaded or produce false accusations. The FTC’s Voice Cloning Challenge likewise frames mitigation across prevention/authentication, real-time detection, and post-use evaluation rather than one perfect classifier.

Safety, abuse resistance, and provenance

Voice synthesis can improve accessibility, localization, education, games, and assistive communication. It can also enable impersonation, fraud, abusive sexual content, false emergency messages, political deception, harassment, authentication bypass, and reputational harm.

A production funnel can require:

authenticated caller and authorized product capability;
voice-selection authorization;
current consent/revocation decision for replica voices;
input content policy;
bounded isolated generation;
waveform and artifact validation;
output/transcript safety checks where appropriate;
watermark/provenance attachment;
publication-time authorization recheck;
usage accounting, abuse reporting, takedown, and audit retention.

Sensitive voice references are biometric-like data. Encrypt them, scope access, minimize retention, prevent cross-tenant retrieval, and do not place raw recordings or embeddings in ordinary logs. Embedding theft may enable impersonation even without the original audio.

Content Credentials can bind assertions about AI generation and edits to an audio asset under the C2PA specification. A watermark may help rediscover provenance after metadata stripping. Neither proves the spoken claim is true, and neither makes unauthorized voice use authorized.

For Mosaic, generated speech is a derived artifact:

source text / approved reference audio
        ↓ normalization + pronunciation releases
acoustic + vocoder + voice-profile releases
        ↓ authorization decision and policy release
generated lossless waveform
        ↓ validation / watermark / transcode
published audio + provenance manifest

If TTS narrates a factual report, the report’s evidence supports the claim. The synthetic voice is a presentation layer, not corroborating evidence.

Performance and capacity

Separate component costs:

resident frontend/acoustic/vocoder/safety weights
+ reference/speaker encodings
+ token and phoneme sequences
+ acoustic activations
+ mel or codec-token buffers
+ vocoder receptive-field state
+ waveform/output encoder buffers
+ streaming queues
+ allocator/runtime overhead
+ concurrency safety margin

Important metrics include text characters or normalized units per second, acoustic frames per second, waveform samples per second, real-time factor, first-audio latency, p95 completed latency, peak device memory, maximum stable concurrent streams, and publication success.

Real-time factor is:

[ \text{RTF} = \frac{\text{generation wall time}}{\text{audio duration}} ]

RTF below one means generation is faster than playback, but it says nothing about first-audio latency or tail stalls. A system can average RTF 0.2 yet wait several seconds before emitting the first chunk.

Batch text/acoustic work by compatible language, model, voice, shape bucket, and configuration. Vocoder batching may conflict with low streaming latency. Enforce per-tenant concurrency and predicted-duration quotas so one long narration cannot monopolize the accelerator.

Cache normalized text or acoustic representations only when the key includes every behavioral input and policy permits reuse. Never let an authorization cache outlive revocation policy.

Failure investigation

Skipped or repeated words

Compare normalized tokens, pronunciation units, duration/alignment coverage, acoustic frames, and ASR transcript. Autoregressive attention can lose alignment; duration models can predict zero or extreme frames; stream assemblers can duplicate sequence numbers. Verify each unit owns a nonempty half-open interval exactly once.

A name is consistently mispronounced

Inspect language selection, normalization, lexicon precedence, phoneme alphabet, grapheme-to-phoneme fallback, stress, and context. Add an approved pronunciation override plus native-speaker regression case. Do not respell the name globally and accidentally change unrelated contexts.

Clicks at chunk boundaries

Check phase/state reset, vocoder receptive-field padding, overlap-add, crossfade, sample-range duplication/gaps, codec frame boundaries, and gain discontinuity. Inspect sample values around the exact boundary and test a continuous tone fixture.

Audio clips after a model update

Compare acoustic/vocoder release, precision, output scaling, loudness stage, encoder, and peak distribution by voice/style. Reject non-finite or above-policy output before encoding. Do not merely clamp and hide systemic saturation.

The voice resembles the wrong person

Quarantine output, preserve restricted artifacts/receipts, compare exact voice profile and model route, inspect cross-tenant cache keys, run similarity against authorized and unintended references, assess published blast radius, revoke the release, and follow notification/takedown procedures.

Treat generation and publication as separate effects. Recheck current authorization at the publication boundary. Mark the generated object quarantined or expired; do not publish because accelerator work already happened.

Seeded output changed after deployment

Compare frontend, lexicon, model, voice profile, duration rules, RNG, precision, device kernels, chunking, postprocessing, and encoder. Preserve the existing artifact when exact replay matters. Seed equality alone is not a complete release.

Alternatives and trade-offs

Approach	Strength	Cost or risk
hosted TTS API	rapid integration and managed scaling	data boundary, provider drift, voice-policy dependency
embedded local TTS	privacy/offline use	platform packaging, memory, quality and safety ownership
designed single voice	simpler identity/consent surface	limited localization and personalization
multispeaker model	shared infrastructure and voice choice	identity leakage and routing complexity
consented voice replica	accessibility and authorized continuity	highest authorization, privacy, abuse, and revocation burden
concatenative/unit selection	predictable approved recordings	storage, limited flexibility, audible joins
recorded human narration	highest intentional performance	time, cost, update latency
earcons/procedural audio	exact non-speech signaling	cannot narrate open-ended language

Do not synthesize when approved prerecorded prompts satisfy the requirement. Authentication codes, emergency warnings, and regulated disclosures may benefit from constrained recorded assets and stronger review.

Exercises

Recall

Why does byte length fail to bound generated duration?
What is the difference between a grapheme, phoneme, acoustic frame, and waveform frame?
Why can a vocoder not repair a skipped word?
What does a speaking-rate change affect that a pitch shift should not?
Why is speaker similarity insufficient for authorization?
What does RTF omit?

Implementation

Add an explicit normalized-token structure with source byte ranges. Support integers from zero to 999 without accepting ambiguous dates or decimals.
Add word-level timing derived from unit ranges and test complete, non-overlapping coverage.
Add stereo WAV output with a typed channel policy and correct block-align/byte-rate arithmetic.
Implement chunked PCM output with sequence numbers and frame ranges. Add a test that reassembly is byte-identical to the unchunked artifact.
Add publication-time authorization revalidation using a separate policy snapshot.
Add a controlled gain stage and prove it rejects rather than hides non-finite input.

Design

Design a consent schema for an accessibility voice that permits private assistive communication but not advertising, resale, model training, or political content.
Design a multilingual evaluation set containing names, dates, units, code-switching, dialect variation, long form, and pronunciation overrides.
Capacity-plan 100 concurrent streaming narrations. State assumptions for predicted duration, first-audio target, acoustic/vocoder RTF, memory, queueing, and cancellation.
Write a runbook for discovering that one tenant heard another tenant’s authorized voice.

Solution guidance

Normalization, pronunciation, pauses, rate, and markup can expand a short string into long audio.
A grapheme belongs to writing, a phoneme to linguistic contrast, an acoustic frame to model features over a window/hop, and a waveform frame to one sample instant across all channels.
It receives an already incomplete acoustic representation; it can render only what is present.
Unit durations and total frames change; pitch frequency should change without changing timing.
Similarity estimates sound-alikeness, not identity authority, purpose, date, or revocation.
It omits startup/first-audio latency, stalls, distribution tails, quality, and publication work.
Preserve the original span, generate an unambiguous word sequence, version the rule set, and reject contexts the bounded grammar cannot interpret.
Aggregate contiguous unit intervals by normalized token and assert the union equals all non-pause spoken frames under a declared pause policy.
Compute block_align = channels × bytes_per_sample; interleave samples and bind channel layout.
Chunks own contiguous half-open frame ranges. No overlap/gap and stable order imply exact reassembly.
Preserve queue-time and publication-time decision receipts; the latter controls the effect.
Validate finite input, calculate gain with bounded arithmetic, measure post-gain peak, and fail if policy would clip.
Make permissions explicit and closed by default; bind voice/profile, subject, product, audience, purpose, retention, dates, revocation, and publication.
Use native reviewers, balanced slices, multiple utterances and speakers, ASR plus listening, and category-specific release floors.
Separate frontend, acoustic, vocoder, encoding, and streaming buffers; reserve for admitted duration buckets and use bounded queues.
Disable the route, quarantine artifacts, preserve access-controlled evidence, inspect tenant scoping/cache keys, assess exposure, notify/take down as required, repair, and add isolation regressions.

Check your understanding

Can you show the normalized spoken form before synthesis?
Can every generated sample be assigned to a contiguous timing plan?
Are text frontend, acoustic model, vocoder, and voice profile independently pinned?
Is unsupported language rejected rather than approximated silently?
Can untrusted users inject SSML, lexicons, URLs, or extreme pauses?
Does replica generation require current purpose-bound authorization?
Is authorization checked again before publication?
Can your WAV be decoded and its declared duration verified independently?
Do clipping, silence, ASR errors, naturalness, and speaker similarity have separate gates?
Can you stop and join a cancelled streaming job without publishing partial success?
Are voice references and embeddings protected from cross-tenant access?
Is synthetic narration represented as derived presentation rather than source evidence?

Practical completion checklist

Authoritative references

W3C, Speech Synthesis Markup Language (SSML) Version 1.1.
van den Oord et al., WaveNet: A Generative Model for Raw Audio.
Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions.
Ren et al., FastSpeech 2.
Kong, Kim, and Bae, HiFi-GAN.
Kim, Kong, and Son, VITS.
ITU-T, P.800: Methods for subjective determination of transmission quality.
ASVspoof, speech deepfake and spoofing-robust verification challenge.
U.S. Copyright Office, Copyright and Artificial Intelligence, Part 1: Digital Replicas.
Federal Trade Commission, Voice Cloning Challenge results.
Coalition for Content Provenance and Authenticity, C2PA Specification 2.2.