Structured Logs, Metrics, Traces, and Redaction

Observability is the ability to ask a new operational question using evidence the running system already emitted. It is not “we have logs.” An AI service needs to distinguish:

• admission rejection from model failure;
• queue wait from model service time;
• first output from final completion;
• one logical run from its attempts, repairs and provider calls;
• software correctness from model quality;
• cached, local and remote inference;
• text/token work from image/audio/video work;
• expected sampling gaps from broken context propagation;
• real incidents from telemetry-pipeline failures.

The companion mosaic-observability crate makes four constraints executable:

1. W3C version-00 traceparent values are strictly validated before use;
2. correlation IDs cannot contain log-control characters;
3. metric stage/outcome labels are closed enums, so cardinality is bounded;
4. sensitive values redact through Display and Debug and cannot be serialized accidentally.

Its runnable example emits a structured JSON stage event and a cumulative histogram/counter snapshot. This is an instrumentation boundary, not a complete vendor backend.

Learning objectives

• choose logs/events, metrics and traces by the question each can answer;
• define a stable telemetry schema and resource identity;
• instrument Tokio futures and spawned work without false parent/child relationships;
• validate and propagate distributed trace context across HTTP, queues and jobs;
• treat incoming sampled flags and baggage as untrusted hints, not authority;
• design counters, gauges and aggregatable histograms with bounded labels;
• correlate high-cardinality run/request identity through traces/logs rather than metric labels;
• capture AI harness/model/tool/usage metadata without recording content by default;
• separate queue, service, first-output and total latency;
• redact at typed construction boundaries and test forbidden-content absence;
• choose head/tail sampling without erasing rare failures or biasing analysis;
• derive service-level indicators for software reliability, quality, cost and freshness;
• keep telemetry loss bounded and visible during overload/shutdown.

Dependencies and mental model

This chapter depends on request identity, structured concurrency, durable run/job IDs, stable error codes, performance stage boundaries and security classification.

Signals answer different questions:

metric
  "Is rejection rate increasing across the fleet?"

trace
  "Where did this sampled run wait, retry, and spend time?"

structured event/log
  "What state transition or policy decision occurred?"

profile
  "Which code stacks consumed resources during the symptom?"

OpenTelemetry models traces, metrics, logs and baggage as separate signals that can share resource and context. tracing gives Rust applications structured spans/events and subscriber composition. Neither chooses safe fields or meaningful boundaries automatically.

Observation, monitoring and debugging

• instrumentation emits typed observations;
• collection/export moves them;
• storage/query retains and aggregates them;
• dashboard/alert/SLO evaluates known conditions;
• exploration asks new questions;
• debugging combines signals with code/config/deploy evidence.

A dashboard is a view, not the source contract. Define instruments and fields in code/schema, review their meaning and evolution, then build queries.

Start with a telemetry contract

Every signal has:

• name and semantic meaning;
• type/unit/temporality;
• required and optional fields;
• field classification and cardinality budget;
• producer/version;
• retention, sampling and access policy;
• owner and operational question;
• compatibility/deprecation plan.

One event might be:

{
  "event_name": "mosaic.stage.completed",
  "service_name": "mosaic",
  "request_id": "request-demo-01",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "stage": "model",
  "outcome": "ok",
  "duration_ms": 87
}

The schema intentionally excludes prompt, completion, media bytes, secret, raw tool arguments and unbounded error message.

Use stable machine codes:

error.type = "provider_rate_limited"
outcome    = "error"
stage      = "model"

Human text can change and may contain data. Do not parse prose to drive alerts.

Resource identity versus event identity

Resource fields describe the producer:

• service name/version/instance;
• deployment environment/region;
• source revision/artifact digest;
• runtime/compiler/target;
• model-serving component where relevant.

Event/span fields describe one operation. Do not repeat mutable build identity manually at every call site; configure it once in the telemetry SDK/export resource.

Version model, harness, prompt, tool schema, retrieval index, evaluator and policy independently. “App version” alone cannot explain AI behavior.

Structured events in Rust

tracing events contain typed fields and occur inside spans. The companion uses a closed function:

pub fn record_stage(
    metrics: &RunMetrics,
    request_id: &CorrelationId,
    trace_parent: TraceParent,
    stage: Stage,
    outcome: Outcome,
    duration_ms: u64,
) {
    metrics.record(stage, outcome, duration_ms);
    tracing::info!(
        event_name = "mosaic.stage.completed",
        service_name = "mosaic",
        request_id = %request_id,
        trace_id = %trace_parent.trace_id(),
        parent_span_id = %trace_parent.parent_id(),
        trace_sampled = trace_parent.sampled(),
        stage = %stage,
        outcome = %outcome,
        duration_ms,
    );
}

The API cannot accept raw prompt/tool output because no parameter exists. That is safer than asking every caller to remember a redaction convention.

Spans are operations, events are moments

A span represents a bounded operation such as:

• HTTP request;
• queue wait;
• durable job attempt;
• retrieval query;
• provider inference;
• tool call;
• output validation/repair;
• artifact persistence.

Events record meaningful points within it: admitted, first token, retry scheduled, policy rejected, lease lost, terminal state committed.

Avoid a span for every token or video frame by default; that can produce enormous volume and overhead. Use counters/events or sampled nested spans at a controlled granularity.

Async instrumentation

Tokio multiplexes tasks across threads, so thread-local log context is insufficient. Instrument the future:

use tracing::Instrument;

tokio::spawn(
    async move {
        // Durable attempt work.
    }
    .instrument(tracing::info_span!(
        "job.attempt",
        run_id = %run_id,
        attempt,
        lease_generation,
    )),
);

Do not hold an EnteredSpan guard across .await; the task may yield and another task can run on the same thread, producing incorrect context. Use Future::instrument, #[instrument], or a framework integration designed for async polling.

#[instrument] records function arguments using structured values or Debug by default. That is dangerous for prompts, headers, secrets and media descriptors:

#[tracing::instrument(
    name = "provider.generate",
    skip(client, prompt, api_key),
    fields(model = %model_id, prompt_bytes = prompt.len())
)]
async fn generate(/* ... */) { /* ... */ }

Review every automatically captured argument. skip_all plus explicit safe fields is often the stronger default at sensitive boundaries.

Library versus application responsibility

Libraries emit spans/events; executables install subscribers/exporters and filtering. A library must not initialize the global subscriber. This lets a CLI choose JSON/stderr, a service export OpenTelemetry, and tests install a capture subscriber.

Subscriber layers may format JSON, filter levels/targets, export spans, bridge log, and add resource fields. Initialize once at process start and fail startup or explicitly fall back according to policy—do not silently run an intended production service with no telemetry.

Distributed context is untrusted protocol input

W3C Trace Context standardizes traceparent and optional tracestate. A version-00 value has:

00-<32 lowercase hex trace-id>-<16 lowercase hex parent-id>-<2 hex flags>

Trace and parent IDs cannot be all zero. The least significant flag bit indicates sampled.

The companion parser enforces exact version-00 length, separator positions, lowercase hex and non-zero IDs:

let parent = TraceParent::parse(
    "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
)?;

assert!(parent.sampled());

It deliberately rejects unsupported versions rather than pretending to be a complete forward-compatible proxy implementation. A production service should use a maintained OpenTelemetry/W3C propagator that implements current processing/version behavior; the narrow parser teaches the trust boundary and provides a test oracle for the accepted subset.

Inbound processing

At an HTTP boundary:

1. enforce header size/count limits;
2. parse supported context;
3. if invalid, ignore/restart according to trust policy—do not return an internal parsing panic;
4. establish server span with a new local span ID;
5. propagate updated context to authorized downstream calls;
6. store trace correlation with the durable job if async continuation is required.

The incoming sampled flag is a recommendation. An attacker can set it on every request to create telemetry load. Apply local sampling and rate policy.

Queues and durable jobs

An HTTP request may finish after admission while a job runs minutes later. Persist a bounded trace link/parent context or create a new job trace linked to the admission trace. Parent/child implies a causal timing relationship; a link may model asynchronous fan-out/retry more honestly.

On retry, create a new attempt span with:

• stable logical run/job ID;
• attempt number and lease generation;
• link to originating/admission trace;
• retry reason code and backoff;
• provider operation/receipt identity when safe.

Do not reuse one span ID across attempts.

Baggage and trust boundaries

Baggage is propagated application metadata, not authentication. Never authorize tenant, capability or quota from baggage. It can cross services and leak through vendors. Allowlist names, limit size/count/value shape, strip at trust boundaries and never place personal/sensitive content there.

Trace IDs themselves enable correlation and have privacy implications. Do not derive them from user identity, IP, time-reversible personal data or secrets.

Metrics: aggregate state with bounded dimensions

Metrics are efficient fleet-wide aggregates when their series set is controlled.

Counter

Monotonically increases:

• accepted/rejected/completed/failed/cancelled runs;
• provider requests/retries;
• input/output token or media-unit totals;
• tool calls;
• bytes stored/streamed.

Compute rates over time. Do not decrement a counter.

Gauge / up-down counter

Current state that rises and falls:

• active workers/runs;
• queue depth;
• in-use permits;
• loaded model/device memory;
• open streams.

A gauge sampled during scrape can miss short peaks. Pair critical transitions with counters or histograms.

Histogram

Records a distribution:

• queue wait;
• service and total duration;
• time to first output;
• input tokens/image pixels/audio/video duration;
• artifact size;
• batch size.

Prefer aggregatable histograms when computing fleet percentiles. Precomputed per-instance quantiles cannot simply be averaged. Bucket boundaries must resolve product thresholds and real distributions; excessively many buckets multiply storage.

The companion uses fixed millisecond bounds:

const LATENCY_BUCKETS_MS: [u64; 8] =
    [5, 10, 25, 50, 100, 250, 1_000, u64::MAX];

Its snapshot exports cumulative counts and represents the final positive-infinity bucket as None. A production exporter maps this to its backend’s histogram model and includes count (and sum when needed). The tutorial implementation uses relaxed atomics: each observation increments one closed counter/bucket, and a concurrent snapshot can see slightly different instants across fields. That is acceptable for monitoring, not an atomic billing ledger.

Cardinality is a resource and reliability limit

Metric series roughly multiply:

metric names × each label's distinct values × replicas/targets

Never label metrics with:

• request/run/trace/span ID;
• tenant/user/email/IP;
• prompt/completion/tool argument;
• URL containing identifiers;
• raw error message;
• artifact/model-generated free text;
• unrestricted model or index revision strings.

Those belong in access-controlled sampled traces/events, or in a bounded catalog mapping.

The companion makes dynamic labels structurally impossible:

pub enum Stage {
    Admission,
    Retrieval,
    Model,
    Validation,
    Persistence,
}

pub enum Outcome {
    Ok,
    Rejected,
    Error,
    Cancelled,
}

It always exposes exactly 5 × 4 = 20 outcome series plus five fixed histograms. Tests assert those bounds. Adding an enum variant is a reviewed cardinality/schema change.

Safe model/provider dimensions

Provider/model can still have many revisions. Options:

• allowlisted model family/tier on aggregate metrics;
• exact revision on trace/resource/deploy manifest;
• separate per-deployment target rather than label;
• top-N/other classification when operationally sufficient.

Do not erase the dimension required to diagnose a rollout, but budget it explicitly.

Correlate without putting IDs everywhere

Metrics find the affected window/service/stage. A trace exemplar can point from a histogram observation to a sampled trace. Trace/event fields hold request/run/attempt identity. Durable state holds the authoritative lifecycle.

The companion CorrelationId accepts 1–64 ASCII alphanumeric, hyphen or underscore bytes. It rejects newline and delimiter-shaped content that could forge a text log:

let request_id = CorrelationId::parse("request-demo-01")?;

Structured JSON encoders should escape values anyway. Input validation is defense in depth and keeps cross-sink identifiers predictable. Do not substitute correlation IDs for authentication or unguessable object authorization.

AI-specific instrumentation

For every logical run:

• modality/task kind;
• harness/prompt/tool-schema/policy revision;
• selected model/provider/execution mode and exact release identity where safe;
• budget requested/consumed;
• input/output usage units;
• cache hit/read/write units;
• retrieval candidate count and selected count;
• tool call count/outcome/error code/duration;
• validation and repair count;
• escalation/fallback route;
• finish/stop reason;
• queue/service/first-output/total duration;
• terminal status and stable error type;
• estimated and reconciled cost;
• eval/feedback linkage outside user-facing hot path.

Do not assume a provider’s “token” definition matches another provider. Retain provider-reported usage and normalized billing units separately.

OpenTelemetry generative-AI semantic conventions evolve and include attributes that may contain sensitive content, especially tool arguments/results and captured model content. Pin/review the semantic-convention version and leave content capture disabled by default.

Text

Measure input/output tokens, context window band, time to first token, inter-token/output rate, finish reason, repair/tool calls and cache. Avoid prompt/completion capture.

Embeddings/retrieval

Measure vector count/dimension band, candidate count, top-k, index/model revision, query/index stage durations and offline retrieval quality. Avoid raw embedding values in telemetry.

Image

Measure dimension/format bands, decode/preprocess/generate/encode stages, steps/batch, preview/final time, peak memory from infrastructure metrics and offline quality. Do not emit image or textual prompt by default.

Audio

Measure duration/sample-rate/channel bands, chunk size, first partial/final time, real-time factor, decoder/model stage and offline WER/task metrics. Transcripts are content.

Video

Measure duration/resolution/frame-rate bands, sampled/decoded/processed/encoded frame counts, per-stage duration, real-time factor and temporary storage. Filenames, paths and frames can be sensitive.

Use bounded bands rather than exact unbounded numeric labels. Exact numeric values belong in histograms/events.

Redaction is an allowlist, not a regex cleanup

Redaction after formatting is fragile:

• JSON/Unicode/encoding/nesting variants bypass patterns;
• secrets can appear in errors, URLs, headers and Debug;
• once queued/exported, deletion may be impossible;
• hashing low-entropy or identifiable content can still enable guessing/correlation.

Classify fields before emission:

public operational
  service version, stage, stable error code

pseudonymous/correlating
  trace/run/request IDs

sensitive content
  prompts, outputs, embeddings, transcripts, media, tool arguments/results

secret/credential
  API keys, authorization headers, cookies, signing material

Default deny sensitive/secret fields. Require an explicit, time-bounded, access-controlled incident mode for any content capture, with user/legal policy, sampling, encryption, retention/deletion and audit.

Typed redaction

The companion wrapper:

pub struct Sensitive<T>(T);

impl<T> fmt::Debug for Sensitive<T> {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        formatter.write_str("[REDACTED]")
    }
}

impl<T> fmt::Display for Sensitive<T> {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        formatter.write_str("[REDACTED]")
    }
}

It intentionally does not implement Serialize. expose() exists for the narrow provider/client boundary. This reduces accidental common-path leakage but cannot stop deliberate exposure, copies inside dependencies, memory dumps or a secret embedded in an ordinary String.

Add negative tests that capture the actual configured subscriber/export payload and assert known sentinel prompt/key/header/media names are absent. Test error paths, debug level, panic reports and HTTP client instrumentation too.

Errors

Map rich internal errors to:

• stable low-cardinality error.type;
• safe stage/outcome;
• chain/source only in a protected diagnostic path after review;
• no raw provider body, SQL, prompt, URL query, header or filesystem path by default.

Never log then return the same error at every layer. That duplicates volume and obscures ownership. Record at the boundary that has action/context, or attach fields to one span.

Sampling without blindness

Head sampling

Decides near trace start, often deterministically from trace ID. Cheap and consistent but cannot know a later error/slow outcome. It can miss rare failures.

Tail sampling

Buffers enough trace data to decide after observing error, latency, deployment or attribute. It can retain valuable anomalies but costs collector memory/latency and needs a policy when collectors overload or traces arrive late/incompletely.

Layered policy

A production policy might:

• retain all bounded metrics;
• retain security/audit events under a separate non-sampled policy;
• head-sample a baseline proportion by valid trace ID;
• tail-retain errors, slow runs, new release canaries and selected rare paths;
• rate-limit per source/tenant to resist forced sampling;
• record sampling rule/rate in metadata;
• keep content disabled regardless of trace sampling.

Sampling changes observed distributions. Weight/interpret sampled counts correctly and never derive billing, quota or authoritative audit from sampled telemetry.

Logs also need rate controls and deduplication. During a provider outage, logging one stack/error body per retry can create a secondary outage and leak responses.

SLOs for an AI system

Software indicators:

• accepted request availability;
• terminal completion/correct rejection rate;
• queue and end-to-end latency distributions;
• streaming first-output latency;
• stale lease/recovery time;
• data/artifact integrity;
• dropped telemetry/export lag.

AI/product indicators:

• task success/eval score by version and slice;
• unsupported/hallucination/grounding policy failures;
• human escalation/correction rate;
• retrieval recall/coverage;
• media quality/accessibility checks;
• feedback with selection-bias caveats.

Efficiency indicators:

• usage/cost per successful run;
• retry/repair/escalation amplification;
• tokens/frames/audio-seconds processed;
• cache effectiveness;
• saturation and queue age;
• resource utilization/energy where available.

Define the population and denominator. “99.9% successful” is meaningless if rejected, cancelled, timed-out or low-quality runs disappear from the denominator.

Quality signals frequently arrive late and require controlled evaluation. Do not alert on raw user feedback alone; it is sparse, selected and adversarially manipulable. Combine software SLOs with versioned offline/online quality gates.

Export architecture and failure policy

application instrumentation
  → bounded SDK buffers/processors
  → local/sidecar/agent collector
  → controlled exporter
  → backend/storage/query

Rules:

• telemetry export never has unbounded application memory;
• user requests do not block indefinitely on telemetry;
• exporter credentials are scoped separately from model/data credentials;
• TLS/auth and egress destinations are allowlisted;
• collector queues/drops/retries/export latency are themselves monitored;
• overload drops lower-priority telemetry predictably;
• audit/security events use a separately engineered durable path if required;
• shutdown has a bounded flush deadline and reports loss;
• development/test defaults cannot silently enable production content capture.

Synchronous logging on a hot async path can block workers. Unbounded async logging moves the crash to memory. Choose a bounded queue, severity policy and loss metric.

Telemetry must degrade safely when the backend is unavailable. The application should usually continue within bounded local resource use; an audit-required transaction may have a different fail-closed contract that must be explicit.

Runnable companion

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-observability --all-targets
cargo clippy -p mosaic-observability --all-targets -- -D warnings
cargo run -q -p mosaic-observability --example structured_signals

The example:

1. installs a JSON tracing-subscriber in the executable;
2. parses a safe request ID and version-00 trace parent;
3. creates a run span;
4. demonstrates a disabled debug event with a redaction type;
5. records one 87 ms model success event;
6. prints a versioned bounded metrics snapshot.

Six library tests and one focused example test prove:

• valid trace context round-trips and sampled is bit-masked;
• all-zero, uppercase and malformed trace context is rejected;
• secret Display and Debug reveal only [REDACTED];
• newline log injection is rejected;
• fixed histograms are cumulative and series count is bounded;
• one stage call correlates the event path and increments exactly one series.

The example JSON omits wall-clock time to remain readable. Production events require an exporter timestamp/observed timestamp and synchronized clock policy.

Failure investigation: metric backend becomes expensive or slow

1. identify new metric names/labels and series growth by deployment;
2. find unbounded label values such as request/user/error/path/model revision;
3. stop or relabel/drop the offending series at a controlled collector boundary;
4. preserve enough schema/deploy evidence to identify the producer;
5. replace the label with a bounded enum/band or move identity to traces/events;
6. add a cardinality budget test/review and backend quota alert;
7. assess telemetry loss and downstream cost.

Deleting old series may not immediately recover stored cardinality cost.

Failure investigation: trace has missing or impossible parentage

1. preserve trace/span IDs, service revisions and raw propagation metadata safely;
2. inspect HTTP client/server and queue producer/consumer instrumentation;
3. check task spawning and whether futures were instrumented;
4. find EnteredSpan guards held across .await;
5. check invalid/header truncation/trust-boundary restarts and sampling policy;
6. model asynchronous work with links where parent/child is false;
7. add an integration test spanning admission, persisted job, retry and terminal event.

Thread IDs cannot reconstruct Tokio task causality.

Failure investigation: a prompt or secret reaches telemetry

1. stop/disable the emitting path or exporter immediately;
2. revoke/rotate credentials if secret material was involved;
3. restrict access and follow backend deletion/retention/incident procedures;
4. identify every sink, replica, buffer, collector and downstream copy;
5. find the construction path—automatic argument capture, Debug, error body, URL/header or dependency instrumentation;
6. replace it with typed allowlisted fields and negative sink-level tests;
7. audit similar paths and incident-mode controls.

Do not copy the leaked content into ordinary tickets or debugging chat.

Failure investigation: telemetry disappears during overload

1. compare application work with SDK queue depth/drop/export error metrics;
2. check collector/backend limits, network and credentials;
3. inspect synchronous blocking and unbounded buffering;
4. preserve high-priority error/audit policy without allowing attacker-forced flood;
5. lower ordinary verbosity/sample rate or shed lower-priority events;
6. restore export and estimate the evidence gap;
7. load-test telemetry with application overload and backend outage.

No drop counter means absence is ambiguous.

Alternatives and trade-offs

Plain text logs

Easy for humans locally; ambiguous parsing/schema and poor aggregation. Keep messages for context but make operational fields structured.

JSON formatted events

Machine-readable and interoperable. Larger, still unsafe if fields are unbounded/sensitive, and a format is not semantic consistency.

Vendor SDK directly

Fast access to backend features. Couples instrumentation to vendor semantics and migration.

OpenTelemetry boundary

Standard propagation/signals/conventions and collector ecosystem. Adds SDK/exporter complexity and evolving semantic conventions; pin and review.

Exact request IDs in metric labels

Easy lookup, catastrophic series cardinality. Use trace exemplars/log search/durable lookup.

Head sampling

Cheap and predictable; misses post-start anomalies.

Tail sampling

Keeps interesting completed traces; buffers data and can fail under overload.

Content capture

Can accelerate narrow quality debugging. Creates major privacy/security/retention/access risk and volume. Default off; controlled, consented, minimal and expiring if ever enabled.

Security and performance implications

Security:

• validate trace/baggage/header sizes and syntax;
• never trust trace/baggage fields for authorization;
• allowlist fields and error codes;
• keep prompt/output/embedding/media/tool content off by default;
• separate audit records from sampled diagnostics;
• scope exporter credentials and egress;
• encrypt access-controlled telemetry and enforce retention/deletion;
• prevent log injection and terminal escape sequences;
• protect correlation IDs from public object-lookup misuse;
• threat-model attackers who force errors, sampled flags and high-volume labels.

Performance:

• measure instrumentation, formatting, allocation and exporter overhead;
• use bounded queues/batches and visible drop counts;
• avoid per-token/frame spans and high-cardinality attributes;
• use histograms that resolve SLO thresholds without excessive buckets;
• do not block async workers on exporters;
• sample traces deliberately but retain aggregate metrics;
• flush only to a bounded shutdown deadline;
• benchmark with production instrumentation enabled.

Exercises

1. Add an Axum layer that parses traceparent, starts a server span, preserves x-request-id, and rejects neither the request nor the process when context is malformed.
2. Instrument durable admission and worker execution as linked traces. Prove retry attempts have distinct span IDs and stable run identity.
3. Extend RunMetrics with queue-wait and first-output histograms while preserving a fixed series budget.
4. Add a bounded model-family label with an other route and a test for unknown revisions.
5. Capture the real JSON subscriber output in a test and assert sentinel API key, prompt, tool argument and filename values never appear.
6. Design a tail-sampling policy for slow/error/new-release traces and state its memory/late-span behavior.
7. Create an SLO dashboard specification covering availability, p95/p99, first output, queue age, eval quality and cost per successful run.
8. Load-test a telemetry-backend outage. Prove application memory stays bounded and drop/export failure signals remain visible.
9. Define safe modality telemetry for a 30-minute customer video without filename, transcript, frame or embedding capture.
10. Version a telemetry-schema migration that renames one error code without breaking a rolling deployment dashboard.

Solution guidance

HTTP middleware should treat invalid context as absent, generate trusted local context through the chosen SDK, and propagate request identity separately. Persist a bounded trace reference on job creation; use span links for asynchronous execution/retries. Histogram additions multiply only by closed Stage; select buckets around explicit UX/SLO thresholds. Model family enters through a validated catalog enum rather than arbitrary provider text.

Sink-level tests install the same formatter/layers into an isolated writer, exercise success and each error path, then search encoded bytes for unique sentinels. Tail sampling needs bounded collector memory, decision timeout, incomplete/late-span policy, per-source rate limits and visible drops. Dashboard definitions name queries, denominators, windows, burn-rate alerts, ownership and runbooks. Backend-outage load tests assert bounded SDK queue and application latency, then verify recovery/flush policy. Video telemetry uses duration/resolution/frame-rate bands and counts/timings; content remains in governed artifacts. Schema migration emits old/new compatibility or updates queries before removal, with a stated deprecation window.

Check your understanding

• Which question is better answered by a metric than a trace?
• Why is a span different from a log event?
• Why must #[instrument] skip sensitive arguments explicitly?
• What is wrong with holding an entered span guard across .await?
• What makes a version-00 traceparent invalid?
• Why is the incoming sampled flag not an instruction?
• When should async job execution use a trace link?
• Why are run IDs forbidden as metric labels?
• Why can histogram percentiles aggregate while per-instance summary quantiles generally cannot?
• What does the companion’s relaxed-atomic snapshot not guarantee?
• Why does hashing a prompt not automatically anonymize it?
• What operational evidence reveals a telemetry outage?
• Why are sampled traces unsuitable for billing/audit?
• Which AI version identities must be recorded independently?

Completion checklist

• Every event/metric/span answers a named operational question.
• Names, units, fields, types, owners and compatibility are documented.
• Resource identity includes release/artifact context.
• Async futures and spawned tasks preserve truthful context.
• Trace context is size/syntax validated and never authorizes access.
• Queue/job/retry propagation has an explicit parent/link model.
• Metrics use bounded reviewed labels and aggregatable distributions.
• Queue, service, first-output and total durations are separate.
• Run/request/tenant/content/error prose is absent from metric labels.
• Prompts, outputs, embeddings, media, tool content and secrets are default-denied.
• Negative tests inspect configured sink output.
• Sampling policy covers rare errors, overload abuse and bias.
• SDK/collector buffers are bounded and drops/export failures are visible.
• Shutdown flush has a deadline and loss policy.
• Software, quality and efficiency SLOs state denominators.
• I ran the six observability tests, focused example test and JSON example.

Authoritative references

• OpenTelemetry: signals
• OpenTelemetry semantic conventions
• OpenTelemetry generative-AI attribute registry
• W3C Trace Context
• Prometheus metric and label naming
• Prometheus histograms and summaries
• tracing crate documentation
• tracing::Instrument for futures
• tracing-subscriber JSON formatting