Day 35 — Freeze the Source

The TypeScript experiment begins by scraping SITE_URL and immediately placing the returned Markdown inside model prompts. That is convenient for discovery, but it combines acquisition and generation. Today you separate them and make the first required run reproducible.

Run before reading

cd rust/rust-ai-engineering
cargo run -p rankforge -- build \
  --source fixtures/rankforge/site.md \
  --responses fixtures/rankforge/responses.json \
  --output target/rankforge

Then change one byte in a copy of site.md, run again, and compare source_sha256 in run-report.json. The digest identifies the evidence used by a run. It does not prove that the website was truthful.

Build the command boundary

Rankforge accepts paths as PathBuf values:

#[derive(Debug, clap::Subcommand)]
enum Command {
    Build {
        #[arg(long)]
        source: std::path::PathBuf,
        #[arg(long)]
        responses: std::path::PathBuf,
        #[arg(long, default_value = "target/rankforge")]
        output: std::path::PathBuf,
    },
}

The pipeline owns the source as Vec<u8>. Hashing borrows &[u8], UTF-8 decoding can borrow a view, and prompt compilation can borrow &str. There is no reason for every stage to clone the entire site.

Acquisition is its own capability

The offline project starts with a file. A live extension can define this boundary:

pub trait SiteAcquirer {
    fn acquire(&self, url: &str) -> Result<AcquiredSite, AcquireError>;
}

pub struct AcquiredSite {
    pub requested_url: String,
    pub final_url: String,
    pub fetched_at: String,
    pub markdown: String,
}

Record redirects, fetch time, content type, and the final bytes. Do not let an adapter return only a string with no origin. The scraper response is an observation with provenance, not trusted instructions for the model.

Break it deliberately

Run with a missing source:

cargo run -p rankforge -- build \
  --source fixtures/rankforge/missing.md \
  --responses fixtures/rankforge/responses.json
echo $?

The error names the failed path and the process exits nonzero. Next create an empty source file. That must fail before any publishable artifact is created.

With a live acquirer, also test redirect loops, oversized bodies, HTML returned where Markdown was promised, timeouts, and pages that contain instructions aimed at an AI crawler.

What you built

You made the changing web an optional input producer and gave the core pipeline stable bytes. Rust ownership now tells you who holds the snapshot, while the hash tells you which snapshot a run used.

Next: Day 36 — Stage the Workflow →.