Project · Forge, the Typed AI CLI

Forge is the laboratory for the whole book. It is intentionally not a chatbot. It runs a versioned task, constrains the result, records the causal path, and lets us test harness behavior without paying a provider or depending on a changing model.

Forge began as the smallest complete vertical slice. It now closes the Harness phase with a capability-scoped source tool and a deterministic exit audit. The advanced policies developed across the Harness chapters—semantic validation, typed authorization receipts, bounded agents, retry ownership, causal trace, strict and forked replay, release rollback, and architecture selection—live in reusable companion crates. The CLI proves the product boundary without hiding those policies inside one oversized binary.

The companion implementation for this chapter lives in:

rust/rust-ai-engineering/

The code in that workspace is authoritative. The snippets here explain its decisions; the test suite proves the current behavior.

Learning objectives

By the end of this chapter, you will be able to:

• explain why a scripted model must come before a network provider;
• separate domain, harness, storage, eval, tool, and CLI responsibilities;
• model task and budget invariants with Rust types;
• implement a generate → verify → targeted-repair loop with a hard stop;
• persist a JSONL trace and replay its terminal state without inference;
• run a case-based evaluation suite and compare reports;
• invoke a root-confined, line-bounded source tool through a typed subcommand;
• compile a content-addressed exit report that binds task, budget, run, replay, tool, and evaluation evidence;
• locate the stronger Harness contracts that replace each first-slice simplification;
• distinguish a passing harness test from a claim about model quality.

What this chapter depends on

You need the mental separation from Model vs Harness vs Product and the Rust ideas in Types, Errors, and Valid States and Traits and Workspaces.

The first milestone below deliberately postponed provider HTTP, streaming, tools inside the model loop, cancellation, and forked replay. That sequencing remains important: a vertical slice should establish one complete behavior before adding new failure surfaces. The completed companion now implements those concepts in focused crates and examples, while Forge retains a small, inspectable command surface.

First principle: remove model variance while testing software

The harness contains ordinary software decisions:

• Did configuration validate before work began?
• Did the verifier return the exact violation?
• Was repair attempted at most once?
• Did a provider failure become the right terminal state?
• Did the trace contain one start and one terminal event?
• Can a stored trace be summarized without a provider?

A live model makes these tests slower, costly, and nondeterministic. It can also hide a harness bug by responding differently on the next run.

Forge begins with ScriptedModel: a queue of predetermined responses or failures. This does not fake the product into appearing complete. It isolates the part under test.

#[derive(Debug, Clone)]
pub enum ScriptedStep {
    Response(ModelResponse),
    Failure(String),
}

pub struct ScriptedModel {
    steps: tokio::sync::Mutex<VecDeque<ScriptedStep>>,
}

Each call consumes exactly one step. An exhausted script is an error. That property makes retry and repair counts observable.

The behavior contract

The first milestone accepts:

1. a task JSON file;
2. a scripted-response JSON file;
3. a destination for the JSONL trace.

It must:

1. validate the task before the first model call;
2. ask the model for a JSON object;
3. enforce output size and required fields;
4. return exact validation violations;
5. perform no more than the configured number of repairs;
6. end in one explicit terminal state;
7. write every important decision to the trace.

The fixture fixtures/forge/task-valid.json is executable product configuration:

{
  "id": "repository-risk-review",
  "objective": "Return one concrete reliability risk and its exact source evidence.",
  "input": {
    "repository": "fixtures/repository",
    "focus": "panic paths"
  },
  "output_contract": {
    "required_fields": ["risk", "evidence"],
    "reject_unknown_fields": true
  },
  "budget": {
    "max_model_calls": 2,
    "max_repairs": 1,
    "max_output_bytes": 4096
  },
  "risk": "low"
}

Notice what is absent: provider name, API key, prompt template, or terminal coloring. The task states behavior. Transports and providers are outside the domain value.

Workspace boundaries

The first slice contains six crates:

crates/
  mosaic-domain/    task, budget, model messages, violations, trace events
  mosaic-harness/   model/verifier/trace traits and the run loop
  mosaic-tools/     capability-scoped source reading
  mosaic-storage/   JSONL trace persistence
  mosaic-evals/     cases, expectations, reports, comparison
  forge-cli/        command parsing and application composition

This follows the Cargo workspace model: members share one lockfile and target directory, while cargo ... --workspace operates across the whole companion.

The dependency direction matters:

forge-cli ───────▶ harness ───────▶ domain
    │                 ▲
    ├─▶ storage ───────┘
    └─▶ evals ────────────────▶ domain

tools ────────────────────────▶ standard library

mosaic-domain knows nothing about Clap, Tokio, files, HTTP, or a provider. forge-cli composes capabilities but does not implement the run state machine.

Invariants before inference

RunBudget::validate establishes two important properties:

impl RunBudget {
    pub fn validate(&self) -> Result<(), DomainError> {
        if self.max_model_calls == 0 {
            return Err(DomainError::ZeroModelCalls);
        }
        if self.max_repairs >= self.max_model_calls {
            return Err(DomainError::RepairBudgetExceedsCallBudget);
        }
        Ok(())
    }
}

Why must repairs be fewer than model calls? A repair consumes another model call. A configuration with one call and one repair promises work it can never perform.

This check happens before RunStarted is written. Invalid configuration is not a failed model run; it is a rejected request.

The task also rejects:

• an empty identifier;
• an empty objective;
• an empty required-field name;
• zero model calls.

Later milestones will add time, token, tool, and money budgets. Add each budget only when the runtime can enforce it.

Capability traits

The harness needs three replaceable capabilities:

#[async_trait]
pub trait Model: Send + Sync {
    async fn infer(&self, request: ModelRequest) -> Result<ModelResponse, ModelError>;
}

pub trait Verifier: Send + Sync {
    fn verify(&self, task: &Task, content: &str)
        -> Result<VerifiedOutput, Vec<Violation>>;
}

pub trait TraceSink: Send + Sync {
    fn record(&self, event: &TraceEvent) -> Result<(), TraceSinkError>;
}

The model is async because remote and local inference can wait on I/O or device scheduling. The first verifier is synchronous because parsing a small JSON object needs no async resource. The trace trait is synchronous in this milestone so an event is durably ordered before the next decision.

That last decision has a cost: flushing every JSONL line limits throughput. It is correct for a teaching CLI. A production event store will later batch or transact events without weakening ordering.

Verification is layered

JsonContractVerifier currently establishes:

1. output is within the byte budget;
2. output parses as JSON;
3. the top level is an object;
4. every required field exists;
5. no unknown field exists when strict mode is enabled.

It returns a list rather than the first error:

{
  "path": "$.evidence",
  "code": "missing_required_field",
  "message": "required field `evidence` is missing"
}

This is structural verification, not truth. The string src/lib.rs:12 might cite the wrong line. Later, a semantic verifier will resolve the path, check the line range, and decide whether the source supports the claim.

Do not write “the output is verified” without naming the layer.

The bounded repair loop

The run loop lives in mosaic-harness, not the CLI:

for attempt in 1..=task.budget.max_model_calls {
    let response = model.infer(request).await?;

    match verifier.verify(&task, &response.content) {
        Ok(output) => return Ok(output),
        Err(violations) if repair_attempts < task.budget.max_repairs => {
            repair_attempts += 1;
            feedback = violations;
        }
        Err(violations) => {
            return Err(RunError::InvalidOutput { violations });
        }
    }
}

The real implementation also writes events around every transition. The model receives the exact violations in the next ModelRequest.

The loop does not:

• tell the model vaguely to “try again”;
• silently expand the call budget;
• retry provider failures;
• accept the best-looking invalid output;
• run until the model declares itself finished.

Those omissions are deliberate. Provider retry and fallback need their own policy because they have different idempotency and cost semantics from output repair.

Trace as a causal record

The successful fixture writes four events:

run_started
model_requested
model_responded
run_completed

A repaired run adds:

verification_failed
model_requested
model_responded

Every event contains the run identity. Response events store usage and content size, not raw content. The completion event stores the verified structured output for replay in this local milestone.

A production redaction policy will distinguish:

• safe metadata;
• protected request or response references;
• encrypted payload storage;
• information that must never be retained.

Build and run it

From the repository root:

cd rust/rust-ai-engineering

Run the successful fixture:

cargo run -p forge-cli -- run \
  --task fixtures/forge/task-valid.json \
  --responses fixtures/forge/responses-valid.json \
  --trace target/forge-valid.jsonl

Expected result:

{
  "evidence": "src/lib.rs:12",
  "risk": "A user-controlled value reaches panic!."
}

Replay the trace:

cargo run -p forge-cli -- replay \
  --trace target/forge-valid.jsonl

The replay command performs no model call. It verifies:

• the trace begins with run_started;
• every event belongs to the same run;
• exactly one terminal event exists;
• nothing occurs after the terminal event.

Then it reports the terminal state, model-call count, repair count, and event count.

Run the evaluation suite

cargo run -p forge-cli -- eval \
  --suite fixtures/forge/eval-suite.json \
  --output target/forge-eval.json

The seed suite contains four behaviors:

Case	Expected behavior
valid first response	completes in one call
one targeted repair	fails structure once, repairs, then completes
invalid without repair	terminates as invalid_output
provider failure	terminates as model_failed

This is a harness conformance suite. Four passing cases do not mean the future repository-risk model is accurate. They prove the software handles these four controlled trajectories.

Compare two reports:

cargo run -p forge-cli -- compare \
  --baseline target/forge-eval.json \
  --candidate target/forge-eval.json

The command reports improved, regressed, unchanged, missing, and new case IDs. Later it will add quality, latency, cost, and severe-failure gates.

Why stdout and stderr are separate

The binary prints a successful result to stdout and diagnostics to stderr:

match execute(Cli::parse()).await {
    Ok(output) => println!("{output}"),
    Err(error) => eprintln!("forge: {error}"),
}

That makes this useful:

forge run ... | jq '.risk'

Progress spinners, trace notices, and warnings must not corrupt machine-readable output. Streaming will preserve the same distinction.

Clap’s derive API maps parser, subcommand, and argument types into the command tree. Doc comments become help text, so command documentation stays near its typed definition.

Failure investigation

“Scripted model has no response remaining”

The harness made more calls than the fixture supplies. Inspect:

1. max_model_calls;
2. max_repairs;
3. each verification_failed event;
4. the number of scripted response steps.

Do not add arbitrary responses. Determine why the extra transition occurred.

“Repair budget exhausted”

Open the trace and find the last verification_failed event. If the same violation repeats, the repair instruction may not expose the relevant contract, or the model may not be capable. If violations change, the repair is making progress but the allocated budget may still be wrong.

Replay rejects multiple terminals

The runtime or storage layer wrote after a terminal state. Treat this as a state-machine bug. Do not make replay ignore the later event.

Eval passes when the desired answer is wrong

The seed grader checks terminal status, not task correctness. Add a deterministic semantic expectation for the field, citation, or environment state. A grader only proves what it actually checks.

Security and performance implications

The source reader:

• canonicalizes its root;
• rejects absolute paths;
• prevents parent traversal;
• skips symlinks during recursive search;
• limits file bytes, line count, and match count.

Forge exposes it through a typed tool-read subcommand:

cargo run -p forge-cli -- tool-read \
  --root fixtures \
  --path forge/task-valid.json \
  --start-line 1 \
  --end-line 20

The command fixes a one-megabyte file limit and a 200-line read limit. --path is relative to the canonical --root; an absolute path, parent escape, symlink escape, directory, oversized file, or invalid line range fails closed. The returned value is a serialized SourceSlice, not an instruction.

The simple Harness::run command does not grant this tool to its scripted model. The full typed authorization, execution, observation, and receipt pipeline is exercised by:

cargo run -q -p mosaic-tools --example authorized_tool_receipt
cargo run -q -p mosaic-tools --example untrusted_tool_observation

This boundary is deliberate. A CLI being able to invoke a tool does not mean every task or model may invoke it. A capability value—not a prompt sentence—must authorize access.

The current JSONL sink flushes each event. This maximizes crash visibility and simplifies ordering, but it is not a high-throughput event store. Benchmark before replacing it. A batched sink must still prove which events are durable when the process dies.

The task file is user input. It is deserialized and then explicitly validated. Deriving Deserialize does not establish domain invariants.

Exercises

Recall

1. Why is a scripted model more useful than a live provider for testing the repair budget?
2. Which four levels of verification are relevant to the risk report?
3. Why does invalid task configuration occur before run_started?

Implementation

1. Add a max_total_units budget. Count input and output units after each response and fail before a further call would exceed the remaining budget.
2. Add a semantic verifier that requires evidence to match path:start-end, resolves it through SourceReader, and rejects a missing path or out-of-range line.
3. Add a fixture where the first response contains an unknown field and the repair removes it.
4. Add a test that a trace with an event after run_completed fails replay.

Design

1. Should raw model content be stored in the trace, encrypted object storage, or nowhere? Define a policy for local learning, internal enterprise, and regulated environments.
2. Design provider retry separately from output repair. Which failures are retryable? Who owns the combined attempt budget?
3. Design forked replay: which prefix is reused, which event changes, and how is the fork linked to its parent?

Solution guidance

For the unit budget, place accounting in the harness rather than the provider adapter. Provider adapters normalize usage; policy decides whether another call is allowed. Record a budget event before returning failure.

For semantic evidence, parse into a new Citation type rather than repeatedly splitting a string. Resolve only through a root-confined capability. Return separate violations for malformed locator, missing file, invalid range, and unsupported claim.

For forked replay, create a new run ID and store parent_run_id plus fork_event_index. Reuse recorded observations only before that index. Never mutate the original trace.

Check your understanding

1. Why is valid JSON insufficient?

JSON parsing proves syntax. Required fields prove shape. Neither proves the cited source exists, the line supports the claim, or the user authorized access. Those require semantic, evidence, and policy checks.

2. Why is the CLI outside the harness crate?

The CLI is one transport. Keeping command parsing outside the run loop lets a later Axum API and worker reuse identical task, budget, trace, and verification behavior.

3. What does the four-case suite establish?

Only that the current harness reaches the expected terminal state for four deterministic trajectories. It does not measure real model accuracy, generalization, truthfulness, or provider reliability.

Final Harness exit audit

The early commands are individually useful, but the phase needs one reproducible artifact proving that the required capabilities work together.

Run:

cd rust/rust-ai-engineering
cargo test -p forge-cli --example forge_exit_audit
cargo run -q -p forge-cli --example forge_exit_audit

Pinned result:

task SHA-256                 fc4aa5898f737ba4c5a8cb7b2c2c380e1e133e2696b7915a5a3907332ca840f6
model calls observed / cap                                                  2 / 2
repairs observed / cap                                                      1 / 1
output-byte cap                                                               4096
evaluation cases                                                           4 / 4
capability receipts                                                             5
report bytes                                                                 2,159
report SHA-256               e41cb721cc80f35a5f3df2bab8170476bf3d029023f14b01aabb9a614302c9eb

The report is written to:

target/labs/forge-exit/forge-exit-report.json

Two example tests prove the composition and byte-for-byte deterministic serialization.

What the audit actually executes

It does not create a checklist and label every item true. It executes the product:

1. Load and validate task-valid.json.
2. Run the repair fixture with fixed run identity.
3. Observe exactly two model calls and one repair under declared caps.
4. Parse the structured output and require risk and evidence.
5. Persist the full JSONL state transition trace.
6. Replay the trace without a model and recover the terminal counts.
7. Run the four-case harness conformance suite and require 4 / 4.
8. Execute tool-read against the task fixture through a canonical root.
9. Prove the returned source slice contains the declared call budget.
10. Hash every receipt into one final report.

If any command, parser, verifier, budget, trace, eval expectation, root boundary, or pinned report changes unexpectedly, the audit fails.

Five capability receipts

The report gives each claim a supporting digest:

Capability	Executable evidence
Typed task and output contract	Validated task bytes plus accepted strict JSON result
Bounded targeted repair	Two-call trace with one exact structural-repair transition
Provider-free replay	Trace summary reconstructed without invoking Model::infer
Capability-scoped tool	Root-confined SourceSlice under file and line limits
Executable evaluation gate	Four versioned trajectories reach expected terminal states

The receipt hashes are not signatures. They detect mutation relative to a trusted expected report. A production release must authenticate roots through its control plane and provenance policy.

How the completed Harness maps back to Forge

The initial Forge loop is intentionally small. The completed workspace surrounds it with stronger, independently testable contracts:

Requirement	Authoritative implementation and runnable proof
Behavior and completion	mosaic_harness::behavior, behavior_contract_lab
Run task/environment/policy/budget/trace	run_contract, compiled_run_envelope
Prompt and demonstrations	prompt_release, versioned_prompt_program
Provider-neutral multimodal input	provider_request, provider_neutral_multimodal
Context allocation	context_budget, context_budgeter_lab
Retrieval and evidence snapshots	retrieval_contract, snapshot_bound_retrieval
Memory selection and writes	memory_scope, memory_write and their examples
Structured and semantic validation	structured_output, semantic_validation
Typed tools and untrusted observations	mosaic-tools::contract, mosaic-tools::observation
Bounded agent progress and stopping	agent_loop, bounded_agent_loop_lab
Retry/fallback/idempotency	retry_fallback, retry_fallback_replay
Secret-safe causal trace	causal_trace, secret_safe_causal_trace
Strict and forked replay	replay, strict_and_forked_replay
Atomic release and rollback	release, harness_release_rollback
Workflow-versus-agent selection	orchestration_design, workflow_agent_casebook

Each example writes or verifies a content-addressed report and attacks its invariant with deliberate mutations. Forge is the product slice; these modules are the policy library from which a production API, worker, or richer CLI composes.

Why not put every policy in forge run now?

A single command supporting provider routing, multimodal ingestion, memory, arbitrary tools, forking, release rollout, and evaluation would become a framework before its product behavior was known. It would also make failure tests depend on unrelated setup.

The dependency direction instead stays:

typed policy modules
        │
        ▼
small composition boundary
        │
        ├── CLI
        ├── HTTP API
        └── durable worker

The next product adds only the capabilities its behavior specification needs. It imports the tested contracts rather than recreating them.

Final failure investigations

Tool read succeeds outside the intended root

Expected behavior: impossible. The reader canonicalizes the configured root and target and then checks that the target begins with the root. Absolute paths and escaping symlinks are rejected.

If a regression appears, do not add a string-prefix patch. Paths such as /safe/root-other expose why textual prefixes are insufficient. Restore canonical component-aware containment and add the exact escape as a fixture.

Exit audit changes on identical input

Inspect, in order:

1. generated run identities;
2. wall-clock timestamps in serialized output;
3. nondeterministic map iteration;
4. unstable filesystem paths;
5. changing fixture bytes;
6. trace concurrency.

The audit fixes the run ID, excludes transient paths and timestamps from its report, uses typed ordered values, and gives concurrent test invocations independent trace files.

Do not “fix” nondeterminism by updating the expected hash on every run. Find the uncontrolled input.

Budget says two calls but trace reports three

Treat the trace as evidence of a runtime defect. The task budget is a promise enforced before each transition, not metadata displayed after execution. Inspect repair ownership and any provider retry layer. Output repair and transport retry must not silently spend each other’s counters.

The audit passes but model quality is poor

The Forge suite is a harness conformance suite. Scripted results remove model variance so the software can be tested exactly. Real task quality requires the held-out, slice-aware, confidence, human, multimodal, latency, cost, reliability, and safety evaluation engineering in Part 6.

Completion checklist

• cargo run -p forge-cli -- run ... prints the expected structured result.
• replay summarizes the stored trace without a model.
• tool-read enforces canonical root, file-byte, and line-count boundaries.
• the four-case eval suite passes.
• the repair trace spends exactly one repair and two model calls.
• the exit audit composes five capability receipts.
• the exit report reproduces byte-for-byte at its pinned digest.
• the focused tests pass.
• rerun the full workspace format, Clippy, test, example, dependency, and site gates after any local extension.

Authoritative references

• Cargo workspaces
• Clap derive reference
• Serde derive
• Serde JSON
• Tokio testing
• Rust API guidelines

Next milestone

Forge’s Harness-phase acceptance is complete. Part 6 turns product requirements into stronger evaluation fixtures, graders, statistical comparisons, slice gates, shadow decisions, and production-failure regressions. The scripted suite remains the software-conformance base; broader quality evaluation must extend it without weakening deterministic failure coverage.