Harness Versioning, Release Identity, and Rollback

An AI application’s behavior does not come from its binary alone.

Change the prompt, behavior specification, retrieval policy, tool registry, validator, loop budget, model route, or evaluation gate and the product may behave differently even when the executable is unchanged. Deploy only a container tag and you cannot answer the most important incident question:

Which exact behavior-producing system handled this run?

A harness release is therefore an atomic, content-addressed bundle of every component that can change run semantics. A human-readable version describes compatibility intent. The bundle digest identifies exact bytes. Qualification receipts prove which evidence passed. A rollout plan limits exposure. A rollback receipt proves which known-good release was restored and when new admission was fenced.

The reference implementation is mosaic_harness::release. The runnable harness_release_rollback companion creates a baseline and three-component candidate change, qualifies it, promotes it from shadow to canary, observes one safety violation, and compiles a verified rollback to the exact baseline.

Learning objectives

By the end you will be able to:

separate a release name, semantic version, artifact identity, and deployment revision;
enumerate every component that contributes to harness behavior;
produce one canonical release identity instead of independently mutable “latest” references;
distinguish API compatibility, run compatibility, behavioral change, and artifact equality;
bind candidate lineage and rollback target to a known-good baseline;
require held-out eval, strict replay, security, performance, and human approval evidence;
implement shadow, canary, ramp, and active rollout stages with minimum sample and duration gates;
compare candidate signals against a concurrent control and absolute safety limits;
fence new admissions during rollback while allowing existing runs to finish under their pinned release;
handle state/schema migrations that make simple binary rollback unsafe;
investigate partial releases, contaminated canaries, stale observations, and rollback failures;
secure release provenance, approvals, artifacts, and operator authority.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness release
cargo test -p mosaic-harness --example harness_release_rollback
cargo run -q -p mosaic-harness --example harness_release_rollback

Pinned reference result:

atomic components                       10
changed components                       3
qualification outcome            qualified
shadow action                       promote
canary safety violations                  1
canary action                       rollback
old admission fence                       700
new admission fence                       701
mutation cases                         14 / 14
baseline release SHA-256      acc47fded765f6edf738f92a52c57fb7c4d4cc2d1b80a22ed547e694fe5dfc69
candidate release SHA-256     81b175a81d36cc8faa72455cd02bc34456a707f0404866342b7fbaeb76a37adb
qualification SHA-256         f8e1807996d2e5be39c61458c5f4d2dbe56edf98638abc6d0c768b48731f67c1
rollout-plan SHA-256          2c6705f4394fc27fbeb404946ebe8302e3cf46cc17b5bc5b6ac85865522fb2a2
rollback SHA-256              b53e60b70087070252d0e85ca7d553932826a6a0b79c7176e84b0d120a9df824
report bytes                  2,980
report SHA-256                6c0e25ea2a732bfa433d3f3a970865d1f1c413b8ec6c643305b5c521b3824b32

Eleven module tests and two example tests cover bundle completeness and order, stable identity, lineage, exact compatibility coverage, qualification accept/reject decisions, observation windows, release/fence binding, stage progression, safety and replay rollback, forged decisions, invalid rollback attempts, stored receipt mutation, and deterministic report serialization.

Four identities, four jobs

Do not overload a single version string.

Identity	Example	Answers
Release name	`forge-harness`	Which product line is this?
Semantic version	`2.4.0`	What compatibility change does the owner claim?
Manifest digest	`sha256:81b1…`	Which exact behavior bundle is this?
Deployment revision	cluster-specific revision 913	Where and when was it installed?

Semantic Versioning defines MAJOR.MINOR.PATCH in relation to a declared public API. Major means an incompatible API change, minor a backward-compatible feature, and patch a backward-compatible fix. That is useful communication, but 2.4.0 is not a byte identity. The same version can be rebuilt incorrectly, and two environments may attach different configuration to the same container.

A digest is exact but says nothing about compatibility. Two one-byte-different manifests have unrelated hashes even when their public behavior is compatible.

A deployment revision is operational history. It may point to the same release installed in two regions at different times.

Use all four:

name + semantic intent + exact content identity + deployment event

The release unit is behavior

The reference bundle requires ten component kinds:

Binary
BehaviorSpecification
PromptProgram
ContextPolicy
OutputContract
ToolRegistry
LoopPolicy
TracePolicy
ReplayPolicy
EvaluationGate

It can additionally bind model routes and retrieval snapshots. A production system may also need:

model weights, provider snapshot, tokenizer, and chat template;
tool executor and sandbox releases;
authorization and tenant-data policies;
memory read/write policies;
schema migrations and serializers;
feature-flag evaluation rules;
retry/fallback route set;
multimodal decoders, preprocessors, and alignment policy;
cost tables and rate limits;
supported historical replay engines.

The test is simple:

If changing this artifact can change a run’s input, authority, decision, output, side effect, evidence, validation, or terminal status, it belongs in the release or in a separately bound run dependency.

Atomic means no mixed “latest”

This is unsafe:

binary = forge:2.4.0
prompt = prompts/latest
tools = registry/current
policy = control-plane/default

The observed combination may never have been tested. A prompt deploy between model inference and repair could even put one run under two behaviors.

Instead, resolve one immutable manifest before admission:

pub struct HarnessReleaseManifest {
    pub release_name: String,
    pub semantic_version: String,
    pub run_interface_sha256: String,
    pub previous_release_sha256: Option<String>,
    pub rollback_target_sha256: Option<String>,
    pub components: Vec<HarnessReleaseComponent>,
}

The component list is closed, unique, ordered, and content-addressed. The manifest digest becomes the run’s release identity. Component stores may still deduplicate shared bytes, but routing changes the manifest pointer atomically.

Identity requires canonical form

The reference requires components in enum order and serializes strict typed structures. In a production format:

pin schema and canonical-serialization release;
reject duplicate keys and unknown fields;
sort only fields whose semantics are defined as sets;
preserve order where order affects behavior;
validate hashes before publishing the root manifest;
authenticate the root digest with signed provenance or a trusted control plane;
never reserialize an old manifest and claim it retained its original identity.

A bare hash detects changes only when the expected hash is trusted. SLSA’s provenance model is useful here: provenance is an authenticated statement about how an artifact was produced, including inputs and build process. The harness release manifest complements build provenance; it does not replace it.

Release lineage and rollback target

A candidate declares:

previous_release_sha256 = currently qualified baseline
rollback_target_sha256  = exact known-good bundle to restore

The reference requires both to equal the baseline used during qualification. This prevents a candidate built against baseline A from being qualified against B and prevents “rollback” from selecting an untested third release.

Lineage is a directed graph, not just a list of version numbers. Hotfix branches, regional variants, and reverted candidates can create branches. Every promotion decision should bind its parent.

Rollback target must be deployable now

Before canary admission, prove:

every baseline artifact still exists and authenticates;
its secrets/configuration references are valid;
its runtime and data schemas remain supported;
capacity is available;
routing can restore it;
operators and automation have authority;
rollback has been rehearsed within the recovery objective.

A digest in a database is not a rollback plan.

Compatibility is multidimensional

“Backward compatible” can mean several different things:

Dimension	Question
Wire/API	Can existing clients send and decode messages?
Stored state	Can old and new workers read current data?
Run	Can a run started under one release finish correctly?
Replay	Can historical traces still be interpreted and verified?
Tool/effect	Are invocation and receipt contracts unchanged or migrated?
Behavior	Are product outcomes intentionally changed and evaluated?
Operations	Can old and new instances coexist during rollout?

The reference emits one entry for every component:

pub enum CompatibilityDisposition {
    Unchanged,
    BackwardCompatible,
    Migrated,
    Breaking,
}

Each entry binds baseline artifact, candidate artifact, compatibility-test receipt, and migration receipt when migration is required. Coverage must match the exact component kinds in both releases. This prevents an uncomfortable changed component from disappearing from the report.

Unchanged requires equal artifact identities. BackwardCompatible requires different identities and a test receipt. Migrated additionally requires a migration receipt. Breaking is an explicit fact; the default qualification policy rejects it.

Behavior change is not API breakage

A more precise prompt may preserve every schema and intentionally change answers. Call that a behavior change, list its change ID, and measure it. Do not hide it under “configuration only.”

Conversely, a digest change may be behavior-preserving: rebuilt documentation metadata or a canonicalization fix could produce different bytes. Exact identity still changes, while behavioral equivalence remains an evidence-backed claim.

Qualification before production

The candidate is not deployable merely because unit tests pass.

The reference qualification evidence binds:

held-out evaluation report;
historical strict-replay suite receipt;
security regression report;
performance/capacity comparison;
behavior-change IDs;
independent approval receipts.

The policy sets limits for:

failed eval cases
strict replay mismatches
safety violations
p95 latency regression
cost regression
changed component count
breaking compatibility
approval count

Qualification returns a receipt even when rejected. A rejected receipt records every triggered violation, allowing review without turning failure into an exception with missing evidence.

let receipt = qualify_release(
    &policy,
    &baseline,
    &candidate,
    &compatibility,
    &evidence,
)?;

assert_eq!(receipt.outcome, QualificationOutcome::Qualified);

Why limit changed components?

Large releases confound diagnosis. If binary, prompts, tool registry, retrieval, model, policy, and grader change simultaneously, a regression has many plausible causes. A change-count limit is not a mathematical safety guarantee, but it reduces experimental entropy and makes rollback decisions faster.

Emergency security releases may need a different policy. Make the exception explicit, authorized, time-bound, and followed by a complete qualification run.

Eval and replay answer different questions

Held-out eval asks whether the candidate meets product requirements on representative cases.

Strict replay asks whether deterministic behavior reproduces historical runs under the declared release mechanics. A new intentional behavior may produce fork differences but should not create unexplained strict replay mismatches for the same bound release.

Security tests probe authority, injection, exfiltration, tenancy, and sandbox boundaries. Performance tests measure latency, throughput, memory, and cost under compatible load.

Passing one does not substitute for the others.

Release state machine

Use explicit states:

assembled
    │ identity + provenance valid
    ▼
qualified ───── rejection evidence ────► rejected
    │
    ▼
shadow ── clean window ─► canary ── clean window ─► ramp ──► active
    │                      │                 │
    └──── violation ───────┴──── violation ─┴────► rollback
                                                     │
                                                     ▼
                                                  restored

Do not infer state from the percentage of pods running a tag. Persist release, qualification, rollout stage, observation, decision, fence epoch, and rollback receipts.

Shadow, canary, ramp, active

Shadow

The candidate observes representative requests or replays captured inputs but owns no production effects. Compare outputs, decisions, validation, latency, and cost against the active release.

Shadow is especially valuable for AI harness changes because quality can regress without HTTP errors. However, it doubles some compute and may expose protected inputs to another processing path. Apply the same tenant/provider policies, and suppress all effects structurally.

Canary

A small, explicit fraction of eligible new runs uses the candidate. The baseline remains the concurrent control. Pin each admitted run to one release for its entire lifetime.

The reference uses basis points to avoid floating-point ambiguity:

500 basis points = 5%

Route by a stable unit such as tenant, principal, conversation, or run—not independently on every model call. Otherwise one workflow mixes releases and contaminates both behavior and metrics.

Ramp

Increase exposure only after minimum observations and duration pass. Larger stages reveal load, queueing, cache, provider-quota, and rare-input behavior absent from a small canary.

Active

At 100%, continue observing for a completion window. “All traffic routed” is not the same as “release completed.” Long-running jobs, delayed effects, and retention workflows may fail later.

Gate both amount and time

A canary with ten requests is not representative. A canary that runs for a day but sees one request is not representative either.

The reference stage requires:

observation_count >= minimum_observations
&& duration_ms >= minimum_duration_ms

When thresholds are not met, the action is Hold, not promote and not fail.

Choose windows using traffic diversity, work-unit duration, delayed outcomes, and statistical power. For a video pipeline, the stage must last long enough for a complete video to reach terminal validation. For a sparse safety failure, zero observations may provide little evidence even after a long time.

The Google SRE canary guidance emphasizes partial, time-limited deployment and comparison with a control. It also warns that before/after comparisons are confounded by time and that canary metrics must use windows no longer than the canary itself.

Signals: relative and absolute

The reference compares:

error-rate regression;
completion-rate regression;
p95 latency regression;
cost regression;
safety violation count;
strict replay mismatch count.

Relative regressions compare candidate to concurrent control. Absolute limits protect against a control that is also unhealthy.

For example:

candidate safety violations > 0             rollback
candidate replay mismatches > 0             rollback
candidate error regression > 100 bps        rollback
candidate p95 regression > 300 bps          rollback
candidate observations below stage floor    hold
otherwise                                   promote

Use quality and safety outcomes specific to the behavior contract:

unsupported claims;
missing citations;
wrong tool authority;
invalid structured output;
false completion;
failed abstention;
memory contamination;
multimodal citation misalignment;
user correction or escalation;
delayed effect reconciliation.

Statistics belong in the next part

Hard thresholds make the release state machine testable, but production comparisons require uncertainty, multiple-comparison control, slice analysis, and power calculations. Part 6 develops those methods.

For now:

predeclare primary metrics and direction;
compare concurrent candidate/control populations;
stratify by important traffic slices;
exclude known instrumentation corruption;
report counts and intervals, not only point estimates;
do not repeatedly peek and stop without accounting for sequential testing;
retain the raw aggregate receipts needed to recompute the decision.

Admission fencing and run pinning

Rollback has a concurrency problem:

Candidate release is active at fence epoch 700.
A run begins and loads candidate components.
A safety violation triggers rollback.
New requests must use baseline.
The existing run must not suddenly load baseline prompt or tools mid-run.

The reference rollback increments the admission fence from 700 to 701.

Admission performs an atomic check:

expected epoch == current epoch
release identity == route release

After rollback, stale routers holding epoch 700 cannot admit new candidate runs. New runs resolve baseline at epoch 701.

Existing runs use:

InflightRunPolicy::DrainPinnedRelease

They may finish under their original candidate bundle if doing so is safe. If the candidate presents an immediate hazard, cancellation policy may terminate them, but that must be a separate explicit decision with compensation and reconciliation—not accidental mid-run release mixing.

Compiling rollback evidence

A rollback receipt is accepted only when:

rollout plan derives from a qualified candidate;
observation belongs to the candidate, stage, and fence;
decision recomputes exactly from that observation;
action is Rollback;
at least one violation exists;
target is the plan’s exact baseline;
new fence is greater than old fence;
rollback authorization is present.

let decision = evaluate_rollout(&plan, &qualification, &canary)?;
let rollback = compile_rollback_receipt(
    &plan,
    &qualification,
    &canary,
    &decision,
    701,
    authorization_sha256,
)?;

The receipt describes a control-plane transition. The deployment system must still:

fence admission;
route new runs to baseline;
verify baseline health;
drain or cancel candidate runs by policy;
reconcile uncertain effects;
preserve incident artifacts;
publish terminal rollback status.

Kubernetes can roll a Deployment back to a previous or specific revision, but the harness control plane must map that infrastructure revision to the exact behavior manifest. kubectl rollout undo alone does not restore a prompt or remote policy that remained on latest.

Data and schema rollback

Binary rollback is easy only when data compatibility is bidirectional.

Classify every migration:

Migration	Forward deploy	Rollback
Add optional field	usually safe	old reader ignores or rejects by contract
Add required field	needs staged population	old writer may corrupt invariant
Destructive rewrite	high risk	requires backup/inverse or roll-forward
External effect	cannot be “un-sent”	compensation/reconciliation
Embedding/index change	dual-read or versioned index	route to old snapshot
Memory schema change	dual schema/migration receipt	preserve old decoder

Use expand–migrate–contract:

expand readers to accept old and new;
deploy compatible writers;
migrate with receipts and checkpoints;
verify;
contract only after the rollback window ends.

For model and retrieval changes, retain old artifacts and indexes. For tool effects, rely on idempotency, effect receipts, and compensating actions where the domain permits them. Never claim a customer-visible email, payment, or publication was undone merely because traffic returned to the old release.

Failure investigations

“The release says 2.4.0, but two runs differ”

Compare manifest digests, not version strings. Then compare:

component identities;
run envelope release identity;
provider/model snapshot;
retrieval and memory snapshots;
feature flags and tenant policy;
trace and replay receipts.

If two different manifests share one semantic version, fix publication controls and mark the ambiguous version as untrustworthy.

“Only the prompt changed, but strict replay fails”

Strict replay under the original release should still pass. Use forked replay to test the new prompt. If original replay fails, another deterministic dependency drifted or evidence is missing.

Qualification should bind both:

strict replay of historical runs under their original release;
controlled fork/eval results for the candidate.

“Shadow looks good; canary fails”

Possible causes:

shadow suppressed a production-only effect path;
candidate and control populations differ;
cache warming or provider quotas changed;
the quality metric needs delayed human feedback;
shadow output was computed but not consumed downstream;
shared dependencies contaminated the control.

Trace routing, release identity, tenant/slice distribution, cache state, and downstream receipt flow. Do not promote because shadow was “mostly right.”

“Rollback routed traffic back but failures continue”

Check:

admission fence propagated;
stale workers can no longer lease new work;
in-flight candidate runs remain;
queues contain candidate-pinned jobs;
destructive data migrations occurred;
caches or feature flags still reference candidate;
external provider/model route did not roll back;
delayed effects are still reconciling.

Rollback completion requires observed baseline health and candidate drain, not a successful router API response.

“Canary metrics disagree”

Check attribution before choosing the favorable metric:

same window?
same eligible population?
candidate/control identity present on every event?
same terminal-outcome delay?
sufficient observations in each slice?
metric aggregation interval shorter than stage?
overlapping canaries or incidents?

When evidence is contaminated, hold or rollback. Do not silently discard the bad signal.

Security and supply chain

Release infrastructure can authorize code, prompts, tools, and policies across all tenants. Treat it as a privileged security system.

Require:

protected source and review;
reproducible or independently verifiable builds where practical;
authenticated build provenance;
malware, vulnerability, license, and policy checks;
content-addressed immutable artifact storage;
signature and attestation verification at admission;
separate builder, approver, and deployer identities;
least-privilege, short-lived deployment credentials;
two-person approval for high-impact tools or policy changes;
tamper-evident qualification and rollback receipts;
audited emergency override with expiry;
rollback artifacts protected from deletion;
tenant/provider policy checks during shadow traffic;
no production effect authority in shadow/replay workers.

Prevent rollback attacks too. An attacker may try to restore a legitimately signed but vulnerable old release. The rollback target must be both known-good for the incident and still allowed by the current minimum-security policy.

Performance and operational cost

Atomic identity adds small hashing and manifest-resolution cost. Cache immutable verified manifests by digest. Resolve once per run, not before every model call.

Shadowing can nearly double inference cost. Control it with:

sampling by representative strata;
local replay for deterministic stages;
cheaper candidate routes where valid;
strict budgets;
result caching only when identity and privacy permit;
early deterministic validation;
bounded retention of paired outputs.

Canarying adds parallel capacity, observability cardinality, and operator complexity. One active candidate per product line is easier to reason about than overlapping canaries. Measure release lead time, qualification time, rollback time, false-positive holds, escaped defects, and error-budget consumption.

Do not reduce the safety sample below usefulness merely to save inference cost. The release policy should make that trade-off explicit.

Implementation walkthrough

Step 1: assemble baseline and candidate

The companion constructs ten ordered components. Candidate 2.4.0 changes prompt, replay policy, and evaluation gate. It binds baseline 2.3.0 as both parent and rollback target.

Step 2: cover every compatibility edge

The report contains one entry per component. Seven are unchanged; three are backward compatible and have compatibility-test receipts.

Step 3: qualify

Held-out eval, historical replay, security, and performance evidence pass. Two independent approval receipts meet policy. The receipt outcome is Qualified.

Step 4: shadow

One hundred observations and sixty seconds meet the shadow floor. All metrics remain within bounds, so the decision promotes to canary.

Step 5: canary and detect

Five hundred observations reach the canary floor. One safety violation exceeds the zero-tolerance limit. The deterministic action is rollback.

Step 6: fence and restore

The rollback receipt restores the baseline digest, advances admission fence 700 to 701, and records DrainPinnedRelease for existing work.

Step 7: mutation-test the control plane

Fourteen cases cover missing components, stale lineage, incomplete compatibility, breaking changes, eval/replay/latency/approval failures, insufficient evidence, valid promotion, stale fence, forged decision, and receipt mutation.

Exercises

Exercise 1: model-route component

Add ModelRoute to both releases. Change from one pinned local model to a remote provider snapshot. Define compatibility, privacy, latency, cost, and output-quality evidence required for promotion.

Exercise 2: state machine

Implement a typed release-state transition function. Reject skipping shadow, promoting a rejected candidate, observing two candidates at one stage, and completing before the active window.

Exercise 3: tenant-stable routing

Write a deterministic router that assigns eligible tenants to canary by release digest and salt. Prove the same tenant remains in one population and that increasing percentage adds tenants without moving existing canary tenants back to control.

Exercise 4: expand–migrate–contract

Design a memory-record schema change. Write old/new reader and writer compatibility tests, a content-addressed migration receipt, rollback-window rule, and destructive contraction gate.

Exercise 5: delayed quality

Add a quality outcome that arrives 24 hours later. Redesign the stage state so traffic can stop ramping while awaiting terminal quality without discarding already collected operational evidence.

Exercise 6: emergency rollback attack

The operational baseline is vulnerable under a new security minimum. Make rollback choose an approved secure ancestor or roll forward to a hotfix. Prove a signed but denied release cannot be restored.

Solution guidance

Treat provider/data-boundary change as more than ordinary compatibility. Require policy approval and representative eval slices.
Model states and transitions as enums; persist receipts, not mutable booleans.
Hash stable routing key with candidate identity; compare against basis-point threshold.
Keep dual readers through the rollback window and contract only after no old release or run can return.
Introduce an AwaitDelayedOutcome hold state with an explicit deadline and safe traffic ceiling.
Apply current security admission policy to the rollback target before compiling the receipt.

Check your understanding

Why can two builds with the same semantic version be different releases?
Which artifacts belong in a harness behavior bundle?
Why must the root manifest be authenticated?
What is the difference between wire compatibility and behavioral equivalence?
Why cover unchanged components in a compatibility report?
Why do held-out eval and strict replay both matter?
What does a rejected qualification receipt provide?
Why require both observation count and duration?
Why compare against a concurrent control?
Why route a whole run, not each model call?
What does an admission fence prevent?
Why do in-flight runs remain pinned during ordinary rollback?
When is roll-forward safer than rollback?
Why can Kubernetes revision rollback be insufficient for an AI harness?

Exit checklist

When every item is true, “version 2.4.0” stops being a vague label. It becomes an exact, qualified, observable, and recoverable behavior release.