Strings, Bytes, Paths, and Unicode Boundaries

Text, bytes, and paths are different domains. Treating them as interchangeable creates corrupted prompts, panics on Unicode boundaries, unusable filenames, and security bugs in tool paths.

Rust makes the distinctions explicit:

• String and &str are valid UTF-8;
• Vec<u8> and &[u8] are arbitrary bytes;
• PathBuf and &Path represent operating-system paths;
• OsString and &OsStr hold platform-native strings that are not necessarily Unicode.

This chapter turns those types into boundary rules for AI and multimodal applications.

Learning objectives

After this chapter, you will be able to:

• distinguish bytes, Unicode scalar values, and user-perceived grapheme clusters;
• convert untrusted bytes to UTF-8 with an explicit failure path;
• slice and truncate strings only at valid semantic boundaries;
• choose String, str, byte buffers, path types, and OS-string types correctly;
• explain normalization and confusable-character risks without assuming one universal policy;
• preserve raw evidence while creating a normalized retrieval representation;
• avoid lossy path and text conversions in security-sensitive code;
• run a grapheme-aware companion example; and
• design byte, token, and UI limits as separate budgets.

Dependencies and downstream use

This chapter depends on ownership, borrowing, slices, and typed errors. It prepares you for:

• prompt construction and tokenization;
• document ingestion and source coordinates;
• filesystem tools and sandbox confinement;
• OCR and speech transcripts;
• binary image, audio, video, tensor, and model formats;
• HTTP body limits and structured-output validation.

The central pipeline keeps representations distinct:

arbitrary bytes
    │ decode with named charset
    ▼
valid UTF-8 source text ──preserve──▶ evidence artifact + byte offsets
    │ normalize for a stated purpose
    ▼
retrieval/model text ──tokenize──▶ model token IDs

Decoding, normalization, segmentation, and tokenization are four different operations.

Rust strings are UTF-8

String owns valid UTF-8. &str borrows valid UTF-8. Therefore every safe Rust string can be passed to Unicode-aware APIs without revalidating its encoding.

let owned = String::from("incident résumé");
let borrowed: &str = &owned;

str::len returns bytes, not characters:

assert_eq!("é".len(), 2);
assert_eq!("é".chars().count(), 1);

Rust does not allow text[0] because a byte may be only part of a UTF-8 code point and indexing would not have one obvious return type or cost.

Three useful notions of length

For "e\u{301} 👩🏽‍💻":

1. bytes are the UTF-8 storage units;
2. Unicode scalar values are what Rust’s char represents;
3. extended grapheme clusters approximate user-perceived characters.

The first visible é is two scalar values (e plus a combining mark). The developer emoji contains multiple scalar values joined into one grapheme. In the companion example, the whole string has seven scalar values but three graphemes, counting the separating space.

Use the measure appropriate to the contract:

Contract	Measure
request body or database byte limit	bytes
Unicode algorithm over scalar values	chars
cursor movement or UI truncation	grapheme clusters
model context	model-specific tokens
display width in a terminal	terminal-cell width under a chosen width policy

“Maximum 500 characters” is incomplete until the product defines which measure it means.

Decode bytes once at the boundary

Network bodies, files, OCR engines, and native libraries often produce bytes:

fn decode_utf8(bytes: &[u8]) -> Result<&str, TextError> {
    Ok(std::str::from_utf8(bytes)?)
}

This borrows the same allocation when it is valid UTF-8. If the source declares a different encoding, use an explicit decoder and record:

• declared encoding;
• decoder/version;
• replacement policy;
• original content hash;
• whether decoding was lossy.

Do not call String::from_utf8_lossy silently. It replaces invalid sequences with U+FFFD. That may be appropriate for a best-effort log preview, but wrong for evidence, identifiers, hashes, source locations, or security decisions.

If arbitrary bytes are expected—images, compressed audio, model weights—keep them as bytes. They are not “invalid text.”

Safe slicing uses byte boundaries

String ranges are byte ranges that must start and end on UTF-8 boundaries:

let text = "évidence";
assert_eq!(text.get(0..2), Some("é"));
assert_eq!(text.get(0..1), None);

Direct indexing with an invalid boundary panics. Checked get is required for untrusted offsets.

char_indices yields byte offsets paired with scalar values:

for (byte_offset, scalar) in text.char_indices() {
    println!("{byte_offset}: {scalar}");
}

Store what the coordinate system means. A source citation can use byte offsets for exact replay, but the UI may also need line/column or grapheme coordinates. Never label a byte offset “character 12.”

Grapheme-aware segmentation

The standard library intentionally exposes UTF-8 and scalar iteration, not every Unicode text segmentation algorithm. This companion uses unicode-segmentation, whose grapheme behavior follows Unicode Standard Annex #29.

use unicode_segmentation::UnicodeSegmentation;

fn truncate_graphemes(input: &str, maximum: usize) -> &str {
    let end = input
        .grapheme_indices(true)
        .nth(maximum)
        .map_or(input.len(), |(index, _)| index);
    &input[..end]
}

The result borrows the original string and never ends inside a grapheme cluster. This is appropriate for a generic UI preview. Language- and product-specific editing may require tailored segmentation, line-breaking, or display-width rules.

Pin and record the Unicode data/library version when segmentation affects stored offsets or eval expectations. Unicode updates can change boundaries for newly standardized sequences.

Normalization is policy, not cleanup

Unicode can represent visually equivalent text with different scalar sequences. Normalization forms such as NFC and NFD provide canonical transformations; NFKC and NFKD additionally apply compatibility mappings that can change presentation distinctions.

Possible policy:

• retain original decoded source for evidence and exact citations;
• derive an NFC-normalized representation for search;
• lowercase only with a documented locale-insensitive or locale-aware rule;
• record each transformation and its version;
• never assume normalized text is safe or non-deceptive.

Normalization can improve matching, but compatibility normalization may collapse distinctions that matter in code, identifiers, credentials, or legal text. Do not rewrite source evidence in place.

Confusables and invisible characters

Different Unicode scalars can look similar. Bidirectional controls and zero-width characters can alter display without obvious visual evidence. AI-generated code, filenames, tool arguments, and authorization labels are sensitive to this.

Defenses depend on context:

• show escaped or annotated views for code review;
• restrict machine identifiers to a documented profile;
• compare canonical internal IDs, not rendered labels;
• surface mixed-script and bidi-control warnings;
• preserve original bytes for forensic analysis;
• do not claim that normalization removes all spoofing.

Human-visible model output is untrusted presentation data. Escape it for HTML, terminal, Markdown, SQL, shell, and log destinations according to that destination—there is no universal “sanitize” function.

Text is not model tokens

A tokenizer maps text into model-specific token IDs. Token counts depend on:

• tokenizer artifact and version;
• exact bytes/text after normalization;
• special-token policy;
• chat template;
• modality placeholders;
• provider-side transformations.

One grapheme can become several tokens. A long ASCII string can tokenize differently across models. Therefore:

1. enforce an early byte limit for resource safety;
2. decode and validate text;
3. count with the exact target tokenizer;
4. allocate the context budget across instructions, evidence, tools, history, and output;
5. record the tokenizer identity in traces and eval artifacts.

Character count is not a reliable proxy for context capacity.

Paths are not UTF-8 strings

Rust uses Path/PathBuf for filesystem paths and OsStr/OsString for platform-native strings. On Unix, a valid path component may contain bytes that are not UTF-8. On Windows, native path representation has different rules.

use std::path::{Path, PathBuf};

fn artifact_path(root: &Path, relative: &Path) -> PathBuf {
    root.join(relative)
}

Avoid converting paths with to_str().unwrap(). If a UI requires text:

• reject non-Unicode paths explicitly, or
• show a clearly labeled lossy display while retaining the original PathBuf.

Do not use the lossy display string for access-control comparisons.

Confinement is a path operation

Checking that a string does not contain ".." is insufficient. Safe root confinement must consider absolute paths, normalization, symlinks, platform prefixes, and time-of-check/time-of-use races.

The companion mosaic-tools::SourceReader:

1. requires a configured root;
2. rejects absolute input;
3. canonicalizes the candidate;
4. verifies the result starts with the canonical root;
5. requires a regular file and applies size limits;
6. skips symlinks during recursive search.

Later security chapters strengthen this for adversarial local environments. The type choice &Path is necessary but not sufficient; authorization and race resistance are separate.

Binary media and text metadata

An evidence envelope should not force every modality into String:

enum EvidenceBody {
    Text(String),
    Binary {
        media_type: String,
        bytes: std::sync::Arc<[u8]>,
    },
    Artifact {
        media_type: String,
        path: std::path::PathBuf,
        content_hash: ContentHash,
    },
}

MIME types and filenames are metadata, not proof of content. Verify magic/container structure with a bounded decoder. A .png path can contain arbitrary bytes; an image/png header can lie.

For huge media, an artifact reference can prevent large buffers from moving through every layer. The content hash and provenance record make the reference verifiable.

Runnable companion implementation

The implementation is crates/mosaic-domain/examples/text_bytes_and_unicode.rs.

cd rust/rust-ai-engineering
cargo run -q -p mosaic-domain --example text_bytes_and_unicode
cargo test --workspace --examples

Expected output:

19 bytes, 7 scalars, 3 graphemes; first: é

The exact first glyph may render as one composed visual character even though the source contains two scalar values. The example:

• accepts bytes only through str::from_utf8;
• reports byte, scalar, and grapheme counts separately;
• truncates on an extended grapheme boundary;
• returns a borrowed prefix without allocating;
• tests invalid UTF-8 and a zero-length truncation.

unicode-segmentation is a focused dependency because the standard library does not promise user-perceived character segmentation. The workspace pins its compatible major line in Cargo.lock; release artifacts must retain that lockfile.

Failure investigation: preview panics on multilingual input

Symptom: a preview function using &text[..100] works for ASCII fixtures but panics on a customer transcript.

Likely cause: byte 100 falls inside a multi-byte UTF-8 scalar value.

Investigation:

1. capture a redacted exact fixture and byte length;
2. reproduce with the same normalization path;
3. check text.is_char_boundary(100);
4. determine whether the product limit means bytes, scalars, graphemes, tokens, or display cells;
5. replace unchecked indexing with the matching segmentation algorithm;
6. add combining-mark, emoji-ZWJ, non-Latin, bidi, and invalid-byte cases.

Fix: for UI “characters,” use a grapheme boundary iterator. For a byte budget, cut only at a valid UTF-8 boundary and document that the visual length varies. For model context, use the tokenizer.

Regression gate: the function must not panic on arbitrary valid UTF-8.

Common failure modes

Calling bytes “characters”

This produces incorrect limits and offsets. Name units in variables and types: max_output_bytes, token_count, grapheme_limit.

Silently lossy decoding

Replacement characters destroy exact evidence. Keep the original and record loss when a preview uses lossy conversion.

Lowercasing as universal normalization

Case mapping can be locale-sensitive and change length. Define the retrieval policy and preserve source text.

Treating paths as URL or shell strings

Paths, URLs, and shell command lines have different parsing and escaping rules. Use separate types and avoid the shell for tool execution.

Slicing model-provided offsets

Check ordering, bounds, and UTF-8 boundaries. Also verify the offsets refer to the exact artifact version.

Logging raw control characters

Escape or structure untrusted output so it cannot forge terminal lines or log records.

Security and performance implications

Security:

• apply byte limits before decoding or normalization to prevent expansion attacks;
• bound output after transformations too;
• retain artifact hashes with coordinates;
• never authorize by a lossy path string;
• escape for the concrete output context;
• review confusables and bidi controls in code/tool surfaces;
• treat tokenizer and normalization changes as versioned behavior.

Performance:

• str::from_utf8 validates linearly and can return a zero-copy view;
• .chars().count() and grapheme counting are linear, not constant time;
• repeated nth calls from the beginning can make truncation quadratic;
• cache boundaries only when profiling justifies their memory cost;
• stream decoding for very large text, carrying incomplete byte sequences correctly;
• preserve byte offsets during parsing to avoid repeated substring search.

Unicode correctness is not free, but guessing is usually more expensive once corrupted citations or production panics appear.

Exercises

Recall

1. What does str::len measure?
2. What does Rust char represent?
3. Why can one grapheme contain multiple char values?
4. Why is PathBuf not just a filesystem-flavored String?

Implementation

1. Add a typed ByteOffset and a checked source-span function.
2. Return both the truncated prefix and the omitted grapheme count.
3. Add fixtures for a combining mark, family emoji, Devanagari text, Arabic with bidi controls, and invalid UTF-8.
4. Build a public preview that escapes control characters without changing stored evidence.
5. Add NFC normalization as a derived artifact with a transformation record.

Design

Specify separate limits for:

• HTTP upload bytes;
• decoded text bytes;
• grapheme count in a card title;
• tokenizer context;
• model output bytes;
• stored diagnostic preview.

For each, state when it is measured, the error returned, whether truncation is allowed, and what provenance is retained.

Failure lab

Create an input where:

• byte count differs from scalar count;
• scalar count differs from grapheme count;
• two canonically equivalent strings have different bytes;
• a filename cannot be represented as UTF-8 on your target platform or in a platform simulation;
• a displayed identifier contains a confusable character.

Write assertions for the exact contract rather than for a vague “character count.”

Solution guidance

For a source span, store start and end as byte offsets tied to an artifact hash. Use str::get and return variants for reversed, out-of-bounds, and non-boundary ranges.

For derived normalization, keep (source_hash, transform_name, transform_version, derived_hash). Coordinates in normalized text must not be applied directly to original bytes unless a mapping was constructed.

For a public preview, iterate and escape control scalars with a visible notation. Do not use that preview as the canonical value.

On Unix, tests can construct OsString from bytes through platform extensions. Keep platform- specific code under cfg and define behavior on Windows rather than assuming parity.

Check your understanding

• Is every byte slice valid text?
• Can &text[0..1] panic even when the string is non-empty?
• Why are grapheme clusters still only an approximation of user perception?
• When is from_utf8_lossy acceptable?
• Why must tokenizer identity be part of a reproducible run?
• Can a valid PathBuf always become &str?
• Does Unicode normalization remove all confusable-character attacks?
• Why should source coordinates include an artifact version or hash?

Completion checklist

• I name byte, scalar, grapheme, token, and display-width units explicitly.
• I decode arbitrary bytes at a boundary with a typed policy.
• I use checked string access for untrusted offsets.
• I preserve original evidence before normalization.
• I use path types for filesystem operations.
• I do not authorize or compare paths through lossy display strings.
• I version segmentation, normalization, and tokenization behavior when it affects outputs.
• I ran the Unicode example and all of its failure tests.

Authoritative references

• std::primitive::str
• std::str
• std::path
• std::ffi::OsStr
• Unicode Standard Annex #29: Text Segmentation
• Unicode Standard Annex #15: Normalization Forms
• Unicode Technical Standard #39: Security Mechanisms
• unicode-segmentation documentation