Day 6 — Ingest and Chunk Documents

Today AskDocs turns files into retrievable units. The important feature is not “load documents.” It is that every returned sentence can still tell you where it came from.

Run it

cd rust
cargo test -p askr rag::tests::chunking_splits_paragraphs_and_respects_budget
cargo run -p askr -- --mock --rag askr/sample-docs "Tell me about coffee"

The CLI prints the filenames and scores of retrieved chunks before it prints the mock response.

Make source identity part of the type

pub struct Chunk {
    pub source: String,
    pub text: String,
}

If you store only text, citations become a reconstruction problem later. Keep identity, locator, and content together from ingestion onward.

build_store reads only .txt and .md files. It owns each file’s String, passes &str into the chunker, and creates owned chunk text that survives after the file buffer drops.

Chunk for the next operation

The baseline prefers paragraph boundaries and packs an oversized paragraph word by word. That choice is observable and imperfect. A tiny chunk may lack context; a giant chunk may bury the answer, waste the context budget, and retrieve on unrelated terms.

Change max_chars from 800 to 80, rerun the same question, and record:

• number of chunks;
• winning source;
• whether the winning text contains enough information;
• prompt length.

This is your first AI experiment: one variable, frozen input and question, observable result.

Break it deliberately

Add:

• an empty file;
• a binary file with .txt extension;
• one paragraph longer than the budget with a single enormous token;
• two files with the same filename from different directories.

The small crate handles some cases and exposes limitations in others. The hardened evidence pipeline adds media type, content digest, strict decoding, source-coordinate maps, size limits, and tenant scope.

Completion proof

Your chunk tests should establish ordinary paragraphs, long paragraphs, blank input, stable source identity, and a boundary case at the exact character budget. Explain which allocations are necessary and why returned chunks cannot borrow the temporary file buffer.

Deep reference: Text normalization and source coordinates.

Next: Day 7 — Embeddings and Search →.