Operate Quality, Latency, and Cost

Traditional service health is necessary but insufficient. An AI endpoint can return HTTP 200 quickly while producing unsupported answers.

Operate five dimensions together:

quality · reliability · latency · cost · safety

Trace the run

Use one run ID across:

• inbound request;
• evidence retrieval;
• context build;
• model calls;
• tool calls;
• verifier outcomes;
• storage;
• client events.

With Rust’s tracing ecosystem, spans can carry safe metadata:

#[tracing::instrument(
    skip(harness, task),
    fields(run_id = %task.run_id, program = %task.program_id)
)]
async fn execute(harness: &Harness, task: Task) -> Result<VerifiedOutput, RunError> {
    harness.run(task).await
}

Do not put raw prompts, media, secrets, or private tool output in ordinary logs.

Metrics

Product and quality

• verified completion rate;
• refusal and escalation rate;
• severe failure rate;
• citation support;
• per-slice eval score;
• user correction or override;
• grader disagreement.

Performance

• time to first output;
• end-to-end p50/p95/p99;
• queue wait;
• model and tool latency;
• ingest throughput;
• local model utilization and memory;
• stream disconnect and cancellation.

Economics

• cost per verified completion;
• model cost by route and program;
• retries and discarded candidates;
• context and output tokens;
• local compute utilization;
• expensive fallback rate.

Raw cost per request is misleading if cheap requests fail more often.

Attribute every cost

struct UsageEvent {
    run_id: RunId,
    stage: StageId,
    model: ModelId,
    input_units: UsageUnits,
    output_units: UsageUnits,
    estimated_cost_micros: u64,
    cached: bool,
}

Images, audio, video, and local inference need modality-specific units as well as normalized money and time.

Service-level objectives

Examples:

• 99.5% of accepted read-only runs reach a terminal state;
• p95 time to first progress event under 2 seconds;
• supported-claim precision above 98% on the incident shadow set;
• zero unauthorized external writes;
• 95% of routine cases stay on the small-model route;
• cost per verified routine completion below the program budget.

Tie alerts to user impact and safety, not every model refusal.

Release a harness like code

A release candidate includes:

• immutable program, prompt, schema, tool, router, and grader versions;
• offline eval comparison;
• safety slice result;
• latency and cost benchmark;
• migration notes;
• canary policy;
• rollback target.

Shadow runs compare candidates without affecting users. Canary runs affect a small, controlled slice. Promote only when guardrails hold.

Drift

Watch for:

• input modality and size changes;
• new languages or domains;
• provider model updates;
• retrieval corpus changes;
• tool API changes;
• grader drift;
• increased fallback rate;
• new failure clusters.

Sample real failures into a reviewed inbox. Do not automatically train on them.

Runbook question

For each alert, document:

1. What user impact does it imply?
2. Which traces and slices confirm it?
3. Can one provider, tool, route, or program version be disabled?
4. What is the safe degraded mode?
5. What evidence is needed before restoration?

The best harness fails into a bounded, observable, reversible state.