MP-24 · Replayable Model Loop

MP-24 is the fifth formally packaged Harness Engineering lab. It turns Chapter 13 into a build–break–measure–defend exercise around a provider-free model/tool/verification loop.

Outcome

You will produce a content-addressed report containing:

• one immutable run, behavior, and loop-policy binding;
• one valid four-step state and evidence chain;
• exact model, tool, repair, cost, step, and time accounting;
• eleven stopping and false-completion mutations;
• one independently replayable terminal receipt;
• one stable report artifact.

Run the reference

cd rust/rust-ai-engineering
cargo test -p mosaic-harness agent_loop
cargo run -q -p mosaic-harness --example bounded_agent_loop_lab

Artifact:

target/labs/mp-24/3ad80edb851be81551ea9a20b9ba9d17e5bfcd7f7c97892622b031406d0538df.json

Expected:

terminal                 completed
mutations                11 / 11
ordered steps SHA-256    ff42896e414d2e4d5929ab44ded863e7b867a4a845f91d945bd178179607ab64
loop receipt SHA-256     11a64af69ff66eec8e0af26154f52e15a06a410248c2683d612119046dde871f
artifact bytes           1,896
artifact SHA-256         35713c204d835ee83517dad10ee778afd7682877cfd0bd3f22489d904df57f33

Predict

Before running, fill this table:

Case	First terminal	New verified progress?	May dispatch another effect?
Valid chain	completed	yes	no
Two no-evidence validations	no_progress	no	no
Same semantic action twice	repeated_action	yes, but cycling	no
Finish without receipt	invalid_completion	no	no
Model/tool/repair/cost over bound	matching budget terminal	possibly	no
Completion at deadline	deadline_exceeded	possibly	no
Tool effect unknown	indeterminate_effect	no	no; reconcile
Nonterminal input ends	input_exhausted	possibly	no

Explain why progress and permission to continue are separate decisions.

Trace

Trace:

policy identity
  -> ordered action/state/evidence steps
  -> exact usage accumulation
  -> terminal precedence
  -> final state and evidence-set identities
  -> loop receipt identity

Then mutate one earlier step. Replay must change or reject every downstream identity that depended on it.

Implement

Choose one extension:

• total evidence-receipt bound;
• token reservation and actual-usage commit;
• bounded longer-cycle detection;
• durable action-reservation event;
• resumable terminal policy;
• reconciliation transition from indeterminate to verified outcome.

Add strict serialization, exact boundary tests, one-over-bound tests, replay tests, and stored receipt mutation. If you add a new terminal or precedence rule, version the evaluator release.

Break

First, temporarily count any changed state as progress. Demonstrate that a model-authored timestamp or plan rewrite can keep the loop alive without new evidence.

Second, accept Finish when model output contains done. The false-completion mutation must catch the regression. Restore receipt-backed completion and retain both tests.

Third, add sequence number to action identity. Show how this defeats repeated-action detection. Restore semantic identity.

Measure

Measure 1, 8, 32, 128, and the maximum permitted steps:

• step validation;
• evidence-set insertion;
• stable step serialization;
• SHA-256 computation;
• receipt reproduction;
• bounded cycle lookup;
• durable append if implemented.

Report median, p95, maximum, fixture size, toolchain, hardware, sample count, and source revision. Keep inference and external tool latency out of this control-plane benchmark.

Defend

Review:

• Can untrusted evidence alter loop policy or authority?
• Can two workers reserve the same last tool call?
• Does cancellation leave a remote effect unresolved?
• Can an old completion receipt satisfy a changed behavior specification?
• Can duplicated content receive random receipt IDs and manufacture progress?
• Can the model increase its own repair budget?
• Does a deadline use a suitable clock?
• Which terminals may resume, under which new identity?

Failure investigation

Symptom: after a restart, the loop repeats its last write and later reports success.

Produce:

last verified event prefix
last state and evidence identities
budget/reservation state
dispatch and idempotency evidence
why the response was absent
reconciled external truth
ownership/lease history
premature completion path
containment and repair
proof of non-recurrence

Do not retry the write until reconciliation proves the first attempt had no effect or the same idempotency key guarantees one logical effect.

Evidence to submit

• unchanged reference report and exact SHA-256;
• your versioned policy and extension;
• ordinary, boundary, failure, replay, cancellation, and security tests;
• one false-progress and one false-completion mutation;
• benchmark artifact and interpretation;
• one restart or uncertainty failure investigation;
• a state-transition diagram;
• a design note naming guarantees and exclusions.

Completion rubric

Score 0–2 for state model, progress, budgets, termination, replay, tests, measurement, and security reasoning. A zero in progress, termination, replay, or security reasoning requires revision. Full completion requires at least 13/16 and no zero.

Continue only when another engineer can reproduce why the loop advanced, why it stopped, which resources it spent, and why its completion proof belongs to this exact run.