Day 9 — Stream and Evaluate

Today AskDocs becomes pleasant to use and difficult to fool. Streaming improves perceived latency; evaluation tells you whether retrieval and answers improved.

Stream through one interface

The small crate exposes a callback:

pub type TokenSink<'a> = dyn FnMut(&str) + Send + 'a;

#[async_trait::async_trait]
pub trait LlmClient: Send + Sync {
    async fn complete_streaming(
        &self,
        request: &CompletionRequest,
        sink: &mut TokenSink<'_>,
    ) -> Result<String>;
}

The terminal prints each delta; tests collect deltas into a string. Both use the same model boundary.

Network chunks are arbitrary byte segments, not guaranteed UTF-8 strings or complete server-sent events. Buffer bytes, split complete event lines, then decode. Test a multi-byte character split across two chunks.

Measure two latencies

• time to first text: when the user sees progress;
• time to completed answer: when downstream validation can finish.

A streaming CLI can feel fast while total work and cost are unchanged. Do not confuse UX improvement with system throughput.

Freeze the product eval

Create at least twelve cases:

• four direct-answer questions;
• two paraphrases;
• two answers split across chunks;
• two unanswerable questions;
• one conflicting-source question;
• one prompt-injection passage inside a document.

For every case record expected source IDs, required facts, forbidden claims, and whether abstention is correct. Grade retrieval recall separately from answer citation accuracy.

Run the gates

cd rust
cargo test -p askr

cd rust-ai-engineering
cargo run -q -p mosaic-harness --example snapshot_bound_retrieval
cargo run -q -p mosaic-evidence --example multimodal_retrieval

The first command proves the small product. The later examples show versioned snapshots, diversity, citation contracts, and stricter evidence identities.

What you built

AskDocs ingests real files, retrieves inspectable chunks, produces a bounded context, streams through a replaceable model, and can be judged on frozen cases. You learned lifetimes and slices in search, async and byte buffering in streaming, and AI evaluation by needing to distinguish a fast demo from a correct answer.

Finish with Project 2 Revision →.