Day 33 — Observe Quality and Roll Back

Today Mosaic learns whether it is working after deployment.

Emit structured signals

cd rust/rust-ai-engineering
cargo run -q -p mosaic-observability --example structured_signals
cargo run -q -p mosaic-performance --example capacity_plan

Every run signal should carry safe correlation fields: run, tenant-safe pseudonymous identity, harness release, model route, operation, attempt, terminal state, latency, usage, and artifact references.

Do not log raw prompts, retrieved documents, secrets, or model output by default. Redaction happens before emission, not in a dashboard query.

Observe four dimensions together

• quality: eval pass rate, citation accuracy, abstention, human escalation;
• reliability: failures, retries, terminal states, queue saturation;
• latency: admission, queue, prefill, first text, completion, tools;
• cost/resources: tokens, provider spend, CPU/GPU time, storage, bandwidth.

A faster release that loses citation accuracy is not unambiguously better.

Roll out by immutable release

Deploy a small cohort, compare paired and live guardrail metrics, then expand. Rollback switches traffic to the last known harness release and compatible model/artifact schema. Test downgrade compatibility and database migration constraints before the incident.

Incident drill

Inject a provider latency spike and a candidate prompt that increases unsupported claims. Verify alerts identify different responsible layers and rollback does not discard durable runs.

Completion proof

Build one dashboard or report where quality, p95 latency, failure rate, and mean cost are sliced by harness release. Define automatic stop conditions and a human owner.

Deep references: Structured logs, metrics, traces, and redaction and Observability and cost.

Next: Day 34 — Ship the Capstone →.