Day 23 — Evaluate and Release the Harness

Today Forge stops treating a prompt as the deployable unit. Behavior depends on model route, prompts, tool schemas, policy, context selection, verifier, loop limits, and code.

Run the eval suite

cd rust/rust-ai-engineering
cargo run -p forge-cli -- eval --suite fixtures/forge/eval-suite.json
cargo run -q -p mosaic-evals --example executable_product_requirements
cargo run -q -p mosaic-harness --example harness_release_rollback

Version the release

pub struct HarnessRelease {
    pub model_route: ReleaseId,
    pub prompt_program: ReleaseId,
    pub tool_registry: ReleaseId,
    pub policy: ReleaseId,
    pub verifier: ReleaseId,
    pub loop_policy: ReleaseId,
}

Hash the stable serialization and record the release ID on every run. Rollback selects an earlier immutable release; it does not rebuild “something similar” from current files.

Grade trajectory and outcome

Final-answer grading misses dangerous or wasteful paths. Add trajectory checks for forbidden effects, unnecessary calls, repeated actions, missing receipts, budget violations, and recovery behavior.

Use paired cases when comparing releases. Report regressions by slice: read-only versus write tasks, short versus long context, ordinary versus adversarial, and supported versus unsupported requests.

Release decision

A candidate ships only if required safety invariants remain perfect and quality, latency, and cost stay inside named thresholds. Keep the previous release loadable and test rollback itself.

What you built

Forge is now a typed, authorized, bounded, replayable, evaluated agent. The model contributes planning and generation; Rust owns what actions exist, whether they may happen, what counts as progress, how the run stops, and how the result is reproduced.

Finish with Project 5 Revision →.