Text Normalization, Structure, Language, and Source Coordinates

“Text” is not one representation.

An uploaded artifact is bytes. A decoder interprets those bytes as Unicode scalar values. A reader perceives grapheme clusters. A parser sees lines, blocks, markup, or syntax nodes. A tokenizer creates model-specific pieces. A citation must eventually point back to the original evidence—not only to whichever rewritten string happened to enter a model.

If a pipeline collapses those layers into one String, ordinary transformations break evidence:

a UTF-8 byte-order mark disappears and every source offset shifts;
CRLF becomes LF and every later line has different byte coordinates;
e plus a combining acute accent becomes precomposed é;
Unicode normalization changes byte length and sometimes scalar order;
a lossy decoder inserts U+FFFD and hides which bytes were invalid;
a language detector labels the whole document even though it contains several languages;
model tokens are later mistaken for durable source coordinates.

This chapter implements a strict text boundary in mosaic-evidence. It retains the original byte digest, accepts bounded UTF-8, handles an optional UTF-8 BOM explicitly, rejects UTF-16/UTF-32 BOMs and malformed UTF-8, normalizes line endings and Unicode NFC, builds line/block spans, and records a coverage map from normalized UTF-8 byte ranges back to original artifact byte ranges. Language is a versioned, ranged observation with basis-point confidence—not a field that silently controls policy.

Learning objectives

By the end you will be able to:

distinguish encoded bytes, Unicode scalar values, grapheme clusters, structural spans, model tokens, and rendered glyphs;
explain why no single offset unit serves every layer;
define a bounded decoding policy before allocation/transformation;
handle UTF-8, UTF-8 BOM, UTF-16/32 BOMs, invalid sequences, and empty input explicitly;
explain why lossy decoding is unsuitable for evidence coordinates;
normalize CRLF and bare CR into LF while retaining original-byte coverage;
choose NFC instead of blindly applying compatibility normalization;
state the Unicode-version/release assumptions behind normalization;
explain why normalized string concatenation is not automatically normalized;
preserve a source digest even when two byte sequences normalize to identical text;
build a coverage map across many-to-one normalization;
distinguish a coverage map from an invertible edit script;
validate UTF-8 byte spans without confusing them with grapheme or scalar offsets;
represent line content and terminators separately;
group nonblank lines into reproducible blocks;
state the exact trailing-newline and whitespace policies;
attach overlapping/disagreeing language observations to normalized spans;
distinguish a tag-shaped label from full BCP 47 registry validation;
avoid treating detector confidence as calibrated probability without evidence;
route model/tokenizer work without losing source citations;
debug normalization, structure, and coordinate failures;
extend the pipeline to legacy encodings, markup, code, and resilient quotes safely.

Prerequisites and dependency order

Read the first two multimodal chapters:

One evidence envelope without erasing modality preserves the encoded artifact and marks charset/line count as unverified declarations.
Provenance graphs, hashes, locators, trust, and transformations records the exact parent, locator, implementation release, parameters, and output digest of a normalization step.

Also review Strings, bytes, paths, and Unicode. This chapter turns those language concepts into a production evidence boundary.

Text normalization precedes chunking, embeddings, retrieval, prompting, tokenization, OCR/ASR merging, and citation rendering. If coordinates are already lost, none of those later systems can reconstruct them reliably.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example text_ingestion

The source fixture contains:

UTF-8 BOM
"Incident cafe" + COMBINING ACUTE ACCENT
CRLF
CRLF
"Déploiement stable"
bare CR
"Operator note"
LF

The normalized view is:

Incident café

Déploiement stable
Operator note

The executable output proves:

Property	Result
Original bytes	56
Bytes after BOM removal	53
Normalized UTF-8 bytes	50
Unicode scalars/graphemes	48 / 48
CRLF / bare-CR rewrites	2 / 1
Lines / blocks	4 / 2
NFC changed text	yes
Original source digest	`e091e3c2…c9590a2`
Normalized `café` span	`[9,14)`
Original covering span	`[12,18)`
Original bytes	`63 61 66 65 cc 81`

The six source bytes encode c, a, f, e, and a two-byte combining accent. The normalized view contains the five UTF-8 bytes for c, a, f, and precomposed é. The map does not pretend those coordinate systems are identical.

First principles: identify every text layer

Encoded artifact bytes

The immutable uploaded/fetched/file-imported byte sequence. Its content digest identifies the exact artifact. Offsets into this layer are the durable source coordinates in version 1.

Decoded Unicode scalar sequence

The result of applying one character-encoding decoder. Rust char represents a Unicode scalar value, not an arbitrary code point, byte, grapheme, glyph, or token.

Normalized Unicode string

A derived representation whose canonical-equivalent sequences use one selected Unicode normalization form and whose line endings follow one selected policy. It has new UTF-8 byte offsets.

Grapheme clusters

Approximate user-perceived characters defined by Unicode text-segmentation rules. One cluster can contain several scalars and bytes: an emoji sequence, base plus combining marks, or conjoining letters. Grapheme boundaries are useful mapping units, but typography can still shape several clusters into glyphs.

Structural representation

Lines, paragraphs/blocks, headings, code spans, DOM nodes, PDF pages, or syntax-tree nodes. Structure is format/application-dependent. This chapter implements only line and blank-line-delimited block structure.

Model tokens

IDs/pieces created by one exact tokenizer release. A token can cover part of a word, whitespace plus word, bytes, or multiple characters. Token offsets are derived runtime coordinates—not source coordinates.

Rendered glyphs

Shapes produced by font/layout engines. Bidirectional reordering and shaping can make visual order different from logical scalar order. A byte span is not a screen rectangle.

The safe flow is:

immutable source bytes
  └─ strict decode + LF + NFC (versioned transform)
      ├─ normalized UTF-8 view
      ├─ normalized→original coverage map
      ├─ lines and blocks
      └─ ranged observations
          └─ chunks/tokens/embeddings with their own mappings

Never overwrite the source artifact with the normalized view.

The policy is part of behavior

pub struct TextIngestPolicy {
    pub max_input_bytes: usize,
    pub max_output_bytes: usize,
    pub max_lines: usize,
    pub max_blocks: usize,
    pub allow_utf8_bom: bool,
}

Every bound must be nonzero and below a protocol safety ceiling:

absolute source/output bytes: 8 MiB
absolute lines: 1,000,000
absolute blocks: 250,000

Those ceilings are defense-in-depth, not recommended production defaults. The runnable example uses 4 KiB, 32 lines, and 8 blocks.

Why bound both input and output? Some decoders/normalizers can expand data, future policies may transcode legacy encodings, and downstream structures add memory. A source-byte cap alone does not bound decoded scalars, lines, tokens, parser nodes, or embeddings.

The semantic transform is named:

mosaic-text-normalization-v1+strict-utf8+lf+nfc

That label must be paired with the companion workspace lockfile/toolchain and implementation release in a provenance receipt. A friendly constant alone is not executable identity.

Decode bytes without guessing

The boundary performs these checks in order:

validate the policy;
reject source bytes over max_input_bytes;
hash the exact original bytes;
recognize UTF-32BE/LE and UTF-16BE/LE BOMs and reject them explicitly;
accept or reject the UTF-8 BOM according to policy;
decode with strict UTF-8;
reject an empty decoded body;
reject NUL with its original byte offset.

Why this chapter accepts only UTF-8

RFC 3629 defines UTF-8. Rust’s str::from_utf8 validates the complete byte slice and returns the first invalid location.

Supporting every legacy encoding in the same implicit boundary invites ambiguity. The same bytes can produce different text under Windows-1252, ISO-8859 variants, Shift_JIS, or other encodings. “Detecting” encoding is commonly a heuristic.

If the product needs a legacy encoding:

require a declared/selected encoding under policy;
preserve original bytes and declaration;
run a bounded explicit transcoder;
record encoding label, transcoder release, errors/replacements, and parameters;
produce a UTF-8 derived artifact;
retain a source-coordinate map or state exactly why mapping is unavailable;
reject undecidable/unsupported cases rather than guessing silently.

The WHATWG Encoding Standard specifies interoperable decoder behavior for web encodings and explains encoding labels/BOM handling. Use a conforming library and frozen fixtures; do not write a legacy decoder from memory.

BOM is evidence

An initial UTF-8 BOM (EF BB BF) is permitted only when allow_utf8_bom is true. It is removed from the normalized view, but:

remains in the original artifact/digest;
shifts the first content source offset to 3;
is recorded as had_utf8_bom;
affects source coverage.

UTF-16/32 BOMs return UnsupportedBom { encoding }; they do not fall through to a generic UTF-8 failure. This gives the caller a safe, explicit transcode path.

An internal U+FEFF is not automatically removed. Position changes semantics; only the initial recognized signature follows the BOM policy.

Why no lossy replacement

String::from_utf8_lossy inserts replacement characters. That is useful for display/log recovery, but inappropriate as the authoritative evidence transform because:

distinct invalid byte sequences can collapse to the same string;
source coordinates become ambiguous;
security-relevant malformed bytes disappear behind �;
a later validator may believe the source was valid Unicode;
the output cannot explain the exact decode failure.

The implementation returns valid_up_to and optional invalid-sequence length without echoing source content.

Why reject NUL

Unicode permits U+0000, but many downstream C APIs, log sinks, terminals, parsers, and legacy formats treat it as terminator or special control. Version 1 rejects it at its original byte offset. This is a product contract, not a claim that all controls are invalid Unicode.

Other bidi/control/invisible characters are retained. A later security scanner should produce typed observations; silently deleting them would alter evidence.

Normalize line endings with a map

The first transformation creates an intermediate UTF-8 string and one mapping unit per input scalar or newline sequence:

CRLF → LF, one normalized byte maps to two original bytes
CR   → LF, one normalized byte maps to one original byte
LF   → LF
other scalar → unchanged UTF-8

The loop records:

struct IntermediateUnit {
    normalized_start: usize,
    normalized_end: usize,
    original_start: usize,
    original_end: usize,
}

The offsets in original_* include the optional BOM. Every intermediate byte therefore has source coverage even though the transformed string is shorter.

Do not normalize line endings by replace("\r\n", "\n").replace('\r', "\n") and discard the edit history. The resulting string may look right while every source citation after the first rewrite is wrong.

Apply Unicode NFC deliberately

The Unicode Normalization Forms specification, UAX #15 defines:

Form	Operation	Important consequence
NFD	canonical decomposition	canonical equivalents share decomposed form
NFC	canonical decomposition + composition	canonical equivalents share composed form where possible
NFKD	compatibility decomposition	may erase meaningful presentation distinctions
NFKC	compatibility decomposition + composition	may turn compatibility characters into ordinary sequences

Version 1 chooses NFC. It makes canonical-equivalent sequences comparable while preserving compatibility distinctions such as circled characters, superscripts, mathematical variants, or width forms that NFKC may erase. UAX #15 explicitly warns against blindly applying compatibility normalization to arbitrary text.

Case-folding, whitespace folding, accent removal, punctuation replacement, homoglyph mapping, and language-specific collation are not NFC. If retrieval needs them, create separate derived search keys with provenance; never call them “normalization” without exact policy.

Normalization versioning

UAX #15 defines normalization stability for assigned characters under its rules, while unassigned characters and implementation data versions still require care. Reproducibility should record:

transform protocol/release;
unicode-normalization crate/version;
locked dependency graph;
compiler/toolchain;
Unicode data version where exposed;
source digest;
output digest;
tests from Unicode’s normalization conformance data.

The companion uses unicode-normalization rather than a handwritten algorithm. Its unit fixtures demonstrate canonical composition; a production conformance gate should run the Unicode NormalizationTest.txt corpus for the pinned release.

Normalized strings are not closed under concatenation

UAX #15 notes that two individually normalized strings can produce a non-normalized concatenation because a character near the boundary can compose/reorder with the other side. Therefore:

normalize the complete intended unit;
or use a normalization-aware append operation;
do not normalize chunks independently and concatenate them blindly.

The implementation calculates whole-string NFC as the expected result.

Map normalization through grapheme clusters

Canonical composition can turn multiple source scalars into one normalized scalar. Reordering can also prevent a one-output-scalar-to-one-source-scalar map.

The implementation:

builds the newline-normalized intermediate string with source units;
segments the intermediate string into extended grapheme clusters using unicode-segmentation;
finds the original covering span of all units overlapping each cluster;
normalizes that cluster to NFC;
maps the resulting normalized cluster range to the original covering span;
concatenates cluster results;
independently normalizes the complete intermediate string;
rejects with MappingInvariant unless both results are identical.

The Unicode Text Segmentation specification, UAX #29 defines default grapheme boundaries. Using a library avoids treating every scalar as a user-visible character.

The resulting segment is:

pub struct SourceMapSegment {
    pub normalized: NormalizedSpan,
    pub original: SourceSpan,
}

Coverage, not a fake bijection

original_cover_for([9,14)) returns the smallest source-byte cover among overlapping mapping segments. If a caller selects only part of a composed cluster at a valid scalar boundary, the source cover can be wider than the normalized selection.

That is honest. Canonical composition/reordering may have no one-to-one inverse.

The map is not:

enough to reconstruct the original bytes;
a character-by-character alignment;
a rendered geometry map;
a token offset map;
guaranteed minimal for every sub-grapheme selection;
permission to discard the source artifact.

Preserve original bytes. For a reversible transformation audit, also store a versioned edit script or exact source/output plus transform receipt.

Validate normalized spans

NormalizedSpan uses half-open UTF-8 byte offsets:

pub struct NormalizedSpan {
    pub start: u64,
    pub end_exclusive: u64,
}

A lookup requires:

start < end;
end <= normalized.len();
both endpoints are Rust UTF-8 character boundaries.

Character-boundary validation prevents slicing inside a multi-byte scalar. It does not guarantee grapheme alignment. Citation producers should prefer structural/grapheme/token mappings and expose when a source cover expands.

Typed names prevent passing an original SourceSpan where a NormalizedSpan is required.

Source digest prevents canonical-equivalence collapse

These source strings normalize identically:

"Café"                  // precomposed U+00E9
"Cafe" + U+0301         // decomposed e + combining acute

Their original byte digests differ. The test proves:

assert_eq!(composed.normalized(), decomposed.normalized());
assert_ne!(composed.source_sha256(), decomposed.source_sha256());

This is why normalized text cannot be the primary artifact identity. Deduplicating only the normalized view would erase byte-level evidence, signatures, format history, and possibly security differences.

Use both identities:

source artifact digest → exact bytes
normalization derivation fingerprint → recipe + parent + output
normalized output digest → exact derived UTF-8 bytes

Build lines without hiding terminators

Each line separates content from its optional LF terminator:

pub struct TextLine {
    pub number: u32,
    pub content: NormalizedSpan,
    pub terminator: Option<NormalizedSpan>,
    pub is_blank: bool,
    pub original_cover: SourceSpan,
}

The exact version-1 rules are:

numbering starts at 1;
line content excludes LF;
terminator is [newline, newline + 1);
a trailing LF belongs to the preceding line;
no synthetic extra empty line is created after a trailing LF;
an empty/Unicode-whitespace-only content slice is blank;
original line coverage includes the terminator when present.

For first\n\nsecond\n, there are three lines: nonblank, blank, nonblank. There is not a fourth invented line after the final terminator.

Different editors/languages may count trailing empty lines differently. The important property is not choosing the “universal” convention—it is defining, versioning, and testing one convention.

Structure is derived, not universal

Blank-line blocks work for plain notes and the example:

pub struct TextBlock {
    pub number: u32,
    pub lines: RangeInclusive<u32>,
    pub normalized: NormalizedSpan,
    pub original_cover: SourceSpan,
}

Consecutive nonblank lines form a block. One or more blank lines separate blocks. A block span starts at the first line’s content start and ends at the last line’s content end; internal newlines remain inside the span.

This is not a Markdown/HTML/source-code/PDF parser. Format-specific structure should produce a separate strict AST with:

parser and grammar release;
node type;
normalized/source spans;
parent/child order;
warnings/recovery actions;
limits on depth/nodes/attributes;
preserved raw source;
tested behavior for malformed syntax.

Do not use blank-line chunking as “semantic paragraphs” for every document.

Language is a ranged observation

The common failure is:

document.language = "en";

Real evidence can contain headings, quotations, code, names, multilingual dialogue, transliteration, and OCR noise. A detector is a model/heuristic with domain-dependent calibration.

Version 1 records:

pub struct LanguageObservation {
    pub normalized: NormalizedSpan,
    pub label: LanguageLabel,
    pub confidence_basis_points: u16,
    pub method_release: String,
}

The range must be a valid nonempty normalized span. Confidence is 0..=10_000 basis points. Method release is a bounded printable identifier. Observations may overlap and disagree; the example/test retains disagreement rather than forcing a winner.

Tag-shaped is not full BCP 47 validation

RFC 5646 specifies language tags. The companion’s LanguageLabel deliberately validates only a strict structural subset:

ASCII;
primary alphabetic subtag length 2–8;
later alphanumeric subtags length 1–8;
hyphen separators;
lowercase canonical storage;
total length at most 63 bytes.

It accepts en, fr, and structurally shaped en-us. It does not implement the full ABNF, grandfathered tags, extension/private-use rules, canonical casing, suppress-script/preferred-value processing, or IANA registry validation. The type is named a label and the limitation is tested.

If interoperable BCP 47 behavior matters, use a conforming parser and pin the registry/canonicalization policy.

Confidence is not automatically probability

9_700 means the producer supplied 97.00% on its declared scale. It does not mean 97 of 100 comparable spans are correct unless held-out calibration demonstrates that.

Evaluate language detection by:

language/script/domain/document-length slices;
code-switching and named entities;
OCR/ASR noise;
short/empty/punctuation-only spans;
known/unknown languages;
calibration and abstention;
downstream routing cost;
disagreement with human review.

Language should not directly grant authorization or suppress content. It may select a tokenizer, retriever, spell policy, or model route only with fallbacks and evaluation.

Step-by-step implementation

1. Declare tight bounds

let policy = TextIngestPolicy {
    max_input_bytes: 4_096,
    max_output_bytes: 4_096,
    max_lines: 32,
    max_blocks: 8,
    allow_utf8_bom: true,
};

2. Ingest once

let mut document = TextDocument::ingest(source, policy)?;

This computes the original digest before transformation and returns either a completely validated document or a typed failure. No partially built document escapes.

3. Inspect structure through spans

for block in document.blocks() {
    let text = document.normalized_slice(block.normalized)?;
    println!("block {}: {text}", block.number);
}

Avoid copying substrings into independent strings unless a later artifact requires it. Borrow from the immutable normalized buffer and carry spans.

4. Attach detector output, not truth

document.add_language_observation(LanguageObservation {
    normalized: first_block,
    label: LanguageLabel::parse("en")?,
    confidence_basis_points: 9_700,
    method_release: "fixture-language-detector@sha256:1111".to_owned(),
})?;

In production the detector configuration/model artifact identity belongs in the transformation receipt or observation method identity.

5. Resolve a citation back to source bytes

let cover = document.original_cover_for(normalized_span)?;

Fetch the original artifact by its verified digest and authorized evidence ID, then slice only after checked u64 → usize, bounds, and tenant access. The example prints hex for a local frozen fixture; ordinary logs should not print source content.

6. Add provenance

Publish normalized UTF-8 as a derived text artifact and create a DerivationRecord:

parent: original text envelope, Locator::Whole
role: source_text
transform_id: text-normalization
implementation_release: exact companion source/container release
parameters: digest of TextIngestPolicy + semantic normalization release
deterministic_claim: true within pinned Unicode/toolchain boundary
output: normalized text envelope/digest

Line/block/language observations can be separate derived records or strict annotations referencing the normalized artifact. Do not mutate the acquisition envelope’s declared charset/line count into verified values.

Error taxonomy

Error	Operational meaning
`InvalidPolicy`	unsafe/zero/excessive caller configuration
`InputTooLarge`	reject before decode
`EmptyText`	no content after optional BOM
`Utf8BomForbidden`	producer/policy mismatch
`UnsupportedBom`	explicit transcode path required
`InvalidUtf8`	quarantine/correct source; never insert replacements silently
`DisallowedNul`	downstream ambiguity/control policy
`OutputTooLarge`	transformed representation exceeded budget
`TooManyLines` / `TooManyBlocks`	structural amplification/budget exceeded
`MappingInvariant`	implementation bug or unsupported normalization boundary; fail closed
`InvalidNormalizedSpan`	range/order/bounds/UTF-8 boundary error
`InvalidLanguageLabel`	unsupported structural label
`InvalidLanguageConfidence`	invalid scale
`InvalidMethodRelease`	unauditable detector identity

Metrics should use bounded error categories, not source content, language label cardinality from attackers, or raw invalid bytes.

Failure investigations

Normalized output looks identical but digest changed

Determine which digest changed:

source digest changes for BOM/newline/composition/encoding byte differences;
normalized output digest changes only if normalized bytes differ;
derivation fingerprint also changes for policy/release/parent/output changes.

Compare bytes in a protected diagnostic tool, not rendered text. Fonts can hide different scalars.

Every citation after line 1 is shifted

Look for BOM stripping, CRLF→LF, compatibility/case/whitespace folding, or lossy replacement performed without a map. Rebuild from original bytes with a versioned transform. Do not apply a constant offset; changes are position-dependent.

A span panics while slicing

All external offsets must enter through normalized_slice, which checks bounds and UTF-8 character boundaries. Search for direct text[start..end], unchecked integer conversion, token/scalar offsets mistaken for bytes, or stale spans from another normalization release.

A citation includes one extra combining sequence

The source map returns coverage segments, not a false bijection. Check whether the selection cuts inside a grapheme-derived map segment. Prefer grapheme-aligned citations or display the expanded source cover explicitly.

Blocks differ between services

Compare transform release, Unicode/Rust whitespace behavior, trailing-newline convention, blank-line definition, and input normalized bytes. Never label both outputs block-v1 if policies differ.

`en-US` became `en-us`

Expected for the structural label’s lowercase storage. If canonical BCP 47 casing/registry semantics matter, replace the subset parser with a conforming versioned component; do not add ad hoc exceptions.

Detector reports high confidence for the wrong language

Inspect span length, code switching, script, OCR errors, names/code, model release, calibration set, and distribution. Add an und/abstain policy and downstream fallback. Do not reinterpret raw confidence as calibrated correctness.

UTF-16 text is rejected

Expected. Invoke an authorized bounded transcoding stage with explicit encoding selection and provenance. Never strip the BOM and reinterpret remaining bytes as UTF-8.

Per-cluster NFC does not equal whole-string NFC

The implementation returns MappingInvariant. Preserve source, record the failure, and repair the mapping algorithm with Unicode conformance fixtures. Never emit a source map for output that was normalized under a different boundary than the map assumes.

Security implications

Bound bytes before decode; bound normalized bytes, lines, blocks, structure nodes, and tokens.
Hash/preserve original bytes before any rewrite.
Reject malformed UTF-8 rather than hiding it with replacement characters.
Treat charset/BOM/declarations as input evidence, not trust.
Use conforming pinned decoders for legacy encodings.
Reject or explicitly model NUL/control policies.
Scan bidi controls, zero-width/invisible characters, confusables, mixed scripts, and suspicious escapes without silently deleting evidence. Unicode UTS #39 provides security guidance/mechanisms.
Never assume NFC prevents homoglyph, bidi, markup, prompt-injection, or identifier attacks.
Escape normalized text for HTML/terminal/SQL/JSON context; normalization is not sanitization.
Parse markup/code with format-specific limits and disable dangerous entity/network/resource resolution.
Keep source maps/artifacts tenant-authorized; excerpts can reveal sensitive parent content.
Do not log original/normalized text, invalid bytes, detector input, or arbitrary language labels by default.
Treat language detectors and OCR/ASR text as untrusted model output.
Never let language label alone decide access, safety, rights, identity, or publication.
Bind every span to artifact/normalization identity; a bare [10,20) is unsafe.
Apply deletion/retention to original text, normalized derivatives, maps, tokens, embeddings, caches, and observations.
Fuzz decoding/normalization/span/structure boundaries and run Unicode conformance data.

Performance implications

Version 1 holds source input, intermediate, expected whole NFC, final normalized output, source-map segments, lines, and blocks during construction. Peak memory is therefore several times source bytes. The absolute bound prevents unlimited growth, but large-scale ingestion should measure:

peak resident memory by language/combining-sequence distribution;
UTF-8 validation and NFC throughput;
number of scalars/graphemes/map segments per byte;
lines/blocks and allocation count;
citation lookup cost;
serialized map size;
downstream token expansion.

Possible optimizations:

use quick-check paths when input is already NFC and contains only LF;
stream/hash source bytes, but only if decoder/normalizer/source-map boundaries remain correct;
coalesce adjacent map segments with exactly contiguous linear byte mapping;
store maps in delta/varint form;
memory-map authorized immutable artifacts where appropriate;
keep spans into one normalized buffer instead of copying every line/block;
cache by source digest + exact transform/parameter identity;
batch language detection by compatible spans without erasing IDs;
run expensive parsing/detection under bounded CPU worker pools with cancellation.

Do not optimize by dropping the source map or original bytes. Measure whether a coarser mapping is acceptable for the product’s citation/evidence requirements.

One denial-of-service edge is very long combining sequences. UAX #15 defines stream-safe text considerations. Version 1’s total byte bound limits them but does not impose per-cluster limits. Before exposing hostile multi-megabyte text, add per-grapheme/nonstarter budgets and corresponding fixtures.

Alternatives and trade-offs

Preserve-only, no normalization

Best for exact editors/signature verification, but search/equality/model input sees canonical variants. Most evidence systems preserve bytes and derive a normalized view.

NFKC

Useful for selected search/identifier/security tasks, but can erase compatibility distinctions. Create a separate purpose-specific derived key; do not replace the evidence view.

Lossy UTF-8

Useful for best-effort UI previews, never the only evidence transformation. Mark replacements and retain exact source spans if a preview is needed.

Scalar-level mapping

More detailed but canonical composition/reordering makes alignment complex. Grapheme coverage is a conservative version-1 choice. Domain-specific code editors may require an edit-script/alignment model.

Text quote selectors

Exact text plus prefix/suffix can re-anchor after edits and complements positions. It also leaks content and can be ambiguous. Add it as a typed locator with normalization/revision identity and match policy.

Full language tag/detector framework

Required for internationalized products. It adds registry/canonicalization updates, script/region semantics, calibration datasets, and potentially native/model dependencies. Version 1 teaches the boundary without pretending to implement all of it.

Exercises

Recall

Why are source bytes, normalized bytes, graphemes, and tokens different coordinate spaces?
Why hash source before removing a BOM?
Why reject UTF-16/32 instead of guessing?
Why is lossy UTF-8 inappropriate for authoritative evidence?
What exact line-ending rewrites does version 1 perform?
Why choose NFC over blindly applying NFKC?
Why can normalized strings become non-normalized when concatenated?
What does a source-map segment mean after composition?
Why can a returned source cover be wider than the selected normalized span?
Does UTF-8 character-boundary validation prove grapheme alignment?
How does version 1 count a trailing newline?
Why are blank-line blocks not universal paragraphs?
Why can two language observations overlap/disagree?
What is missing from the subset language-label parser?
Why is detector confidence not automatically calibrated probability?

Implementation

Freeze the example JSON as a golden fixture and verify the source digest/spans.
Add Unicode NormalizationTest.txt conformance tests for the pinned normalization library.
Property-test that every accepted nonempty normalized span returns an in-bounds nonempty source cover.
Property-test that source maps are ordered, gap-free over normalized output, and source-bounded.
Fuzz BOM/UTF-8/newline/combining-sequence inputs under strict memory/time limits.
Add per-grapheme/nonstarter limits and an adversarial long-combining-sequence fixture.
Add explicit Windows-1252 transcoding with original→UTF-8 mapping and a versioned receipt.
Add TextQuote plus prefix/suffix locator and ambiguity result.
Add Markdown structure with parser warnings, node/depth limits, and source spans.
Add source-code structure using a language parser without executing/building the code.
Map exact tokenizer offsets back through normalized spans to original coverage.
Add a conforming BCP 47 parser and IANA registry/canonicalization policy.
Evaluate a language detector on mixed-language, short, code, OCR-noise, and unknown slices.
Persist normalized artifacts/maps/observations atomically with derivation records.
Add deletion tests proving source, derivatives, embeddings, and maps obey policy.

Design and adversarial review

Design text ingestion for HTML where rendered text must cite DOM/source bytes.
Design PDF text coordinates that preserve page, glyph geometry, reading order, and source object.
Decide whether code should normalize Unicode at all before compilation/security review.
Design edits/revisions while keeping citations bound to the correct artifact version.
Threat-model bidi override characters in logs, diffs, tool arguments, and prompts.
Threat-model two canonically equivalent identifiers with different signatures/source bytes.
Design mixed-language routing with abstention and human escalation.
Define a privacy policy for language observations and snippets.
Design a source map for OCR text where characters come from image polygons, not source text bytes.
Decide which transformations belong in a retrieval key versus evidence view.

Solution guidance

They count different units after different transformations; converting without a map loses provenance.
The BOM is part of exact source identity and shifts every later source byte.
Encoding is ambiguous; explicit transcoding is auditable and testable.
Replacement collapses invalid byte distinctions and obscures failure/source coordinates.
CRLF→LF and CR→LF; LF/other scalars remain.
NFC preserves compatibility distinctions while canonicalizing canonical equivalents.
characters at the boundary can compose or reorder.
That normalized segment was derived from and is covered by the stated original byte interval.
Several source scalars/bytes can compose/reorder into one mapped grapheme segment.
No; it only prevents splitting a UTF-8 scalar.
LF terminates the preceding line; no extra final empty line is invented.
Format semantics differ; blank lines are only a declared plain-text heuristic.
Different methods/spans can legitimately disagree, and disagreement is evidence.
Full RFC grammar, grandfathered/private/extension rules, registry and canonicalization.
Calibration must be measured on comparable held-out data.

For the source-map property test, assert:

first normalized start = 0
each prior normalized end = next normalized start
last normalized end = normalized byte length
every normalized span is nonempty and UTF-8-boundary aligned
every source span is nonempty and within original source length
every random valid nonempty normalized scalar-boundary span resolves

Do not assert source spans are monotonic at scalar granularity if a future normalization/alignment supports reordering; define whether the map stores coverage or ordered correspondence.

For legacy transcoding, freeze invalid-sequence behavior. If replacements are allowed for a preview, record each replacement with original byte span and never label the output lossless.

For language detection, build held-out fixtures with multi-label/ranged ground truth and an und/abstain option. Measure both classification and downstream routing outcomes.

Check your understanding

Does a Rust char equal a user-perceived character?
Can two different source digests normalize to the same string?
Is a UTF-8 BOM part of the original digest?
Does the first normalized content byte necessarily map to source byte zero?
Can invalid UTF-8 be safely repaired without recording the bytes?
Is NFC lowercasing?
Is NFKC always semantics-preserving?
Can two NFC strings be concatenated without rechecking normalization?
What unit does NormalizedSpan count?
Why are normalized and source spans separate Rust types?
Can a normalized selection map to a longer source cover?
Does preserving source coverage allow deleting the original bytes?
Does trim().is_empty() define Markdown paragraphs?
Is the final LF represented as its own synthetic empty line?
Does fr on one block label the whole document?
Does 9_700 prove 97% calibrated accuracy?
Can language determine authorization?
Which provenance fields bind the normalization run?
Which additional mapping is needed after tokenization?
Why must invisible/bidi text be observed rather than silently removed?

Completion checklist

Authoritative references

The next chapter applies the same discipline to images: bounded format probing and decode, EXIF orientation, color/alpha policy, verified pixel coordinate spaces, regions, OCR observations, and vision embeddings. Its citations will enter the same provenance graph rather than replacing pixels with captions.