Trace Design, Secret-Safe Capture, and Causal Debugging

“Log the prompt and response” is not a trace design. It is an uncontrolled copy of sensitive data with uncertain retention, access, structure, and diagnostic value.

A useful AI-system trace must answer:

Which exact run, policy, release, request, evidence, and route were involved?
Which decision consumed which earlier receipts?
What changed, who owned it, and what terminal proof was produced?
Which content can an authorized investigator retrieve?
Which fields were intentionally omitted, referenced, or recorded only as present?
Can deletion, retention, and tenant boundaries still be enforced?
Can mutation, deletion, reordering, and receipt laundering be detected?

The trace must not answer unauthorized questions such as “what was the API key?” or “show every user’s raw prompt forever.”

The design in this chapter is:

typed event draft
  + exact capture rule for every field
  + content references for sensitive payloads
  + presence-only credential observation
  + causal parent and receipt dependencies
  + bounded time/size/count policy
                 │
                 ▼
secret-safe causal record
  + redaction receipt
  + previous-record digest
                 │
                 ▼
replayable trace and terminal receipt

The implementation is mosaic_harness::causal_trace; the runnable companion is secret_safe_causal_trace.

Learning objectives

By the end you will be able to:

separate logs, metrics, distributed traces, audit records, and replay traces;
design a versioned event vocabulary around decisions and receipts;
capture safe metadata through an allowlist rather than redact arbitrary strings later;
represent prompts, outputs, evidence, and personal data as access-scoped content references;
record credential presence without credential value or stable secret digest;
produce a per-event redaction receipt;
build a directed acyclic causal graph from parent events and receipt flow;
use backward slicing to debug a terminal decision;
distinguish integrity chains from authenticity and trusted execution;
set field, event, inline-byte, time, access, sampling, and retention bounds;
investigate trace gaps, secret leaks, tenant drift, and false causal conclusions;
explain which trace evidence supports strict replay and which is observational only.

Prerequisites

Complete the run-envelope and agent-loop chapters first. The trace binds their identities and terminal decisions. Complete production observability so W3C trace context, structured metrics, redacted application logs, and request correlation already have clear boundaries. Complete multimodal provenance because content references need restorable source locators and lineage.

Chapter 16 will use the trace for strict and forked replay. This chapter focuses on capturing the right evidence safely.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness causal_trace
cargo test -p mosaic-harness --example secret_safe_causal_trace
cargo run -q -p mosaic-harness --example secret_safe_causal_trace

Pinned reference result:

events                              5
causal ancestors of terminal        4
mutation cases                     13 / 13
policy SHA-256                     e92abe2ca8ae94ad665178723cf5f4d380cd3180a1d17237cba2dc8c2a708147
trace SHA-256                      fa4b64ff34ad6e00a06a8015e77cfb558b643c5b73a10aca7c276191bd8df5cf
record-set SHA-256                 f7d73c49d2b2f4dc5d2dcdede6cf08c40d6d32bcf5214b55355d462a3b024d12
receipt SHA-256                    85cb533056557258063d85544123409a701186f4ad811b33e9b30e33d35a5bfe
terminal-record SHA-256            1586395e79814d71580d30d1c47aaeebf8b59c67d2837283c67f45f8f85ccb18
report bytes                       2,543
report SHA-256                     e29e888d0b212b7d4a54a5e5f7f20aaa6a2081543ae7f5b85e8597a90716b7ea

Eleven module tests and two example tests cover capture-policy validity, content and credential restrictions, secret-shaped inline rejection, exact rule matching, field/event/byte/time bounds, sequence, causal parents, receipt production and consumption, global duplicate rejection, terminal discipline, strict policy decoding, causal ancestors, hash-linked replay, stored mutation, and absence of raw prompt/output/credential strings from the report.

Choose the observability artifact deliberately

Artifact	Primary question	Typical shape
Metric	How often/how much?	bounded numeric time series
Log	What notable local event occurred?	structured event
Distributed trace	Where did a request spend time?	spans and links
Audit record	Who attempted or changed what under which authority?	append-only security event
Replay trace	What exact inputs and decisions reproduce state?	versioned event/receipt chain
Evidence graph	Which source supports a claim?	provenance DAG and locators

One physical record can contribute to more than one system, but its contract must be clear.

A high-cardinality run ID belongs in a trace/log field, not a metric label. A tool approval belongs in an audit record and replay trace. A prompt body belongs in a protected artifact store, not an ordinary log. Aggregate latency belongs in metrics even if a distributed span also records one duration.

Trying to make one sink satisfy every purpose usually creates expensive, sensitive, unqueryable data.

Start from questions, not available fields

Before defining events, write diagnostic questions:

Why was route B selected?
Which evidence receipts reached the model request?
Did validation consume the output that the model actually produced?
Which policy authorized the tool call?
Where did the retry delay come from?
Why did completion fail after an apparently successful tool result?

Then identify the minimum joins needed:

run envelope
policy and release
event and parent
input receipt identities
output receipt identities
route/action identity
typed decision/result code
bounded usage and time

Do not start with “serialize the request struct.” Internal structs contain convenient but unsafe fields and change for reasons unrelated to the trace contract.

Event vocabulary

The reference uses a closed set:

pub enum CausalEventKind {
    RunStarted,
    ModelDispatched,
    ModelObserved,
    ToolAuthorized,
    ToolObserved,
    ValidationCompleted,
    RetryDecided,
    StateCommitted,
    RunTerminal,
}

The actor is also closed:

Harness | Scheduler | ModelAdapter | ToolExecutor | Validator | Persistence

Closed enums prevent user/model strings from becoming unbounded categories and force schema review when a new semantic event is introduced.

Each event carries:

contiguous sequence;
stable event ID;
event time inside the run trace window;
actor and kind;
earlier parent event IDs;
input receipt identities;
newly produced receipt identities;
typed captured fields;
a redaction receipt;
previous-record identity.

Record decisions, not narration

Prefer:

kind = retry_decided
result = fallback
policy = <digest>
previous_attempt = <receipt>
selected_route = private-fallback

over:

message = "The model seemed slow, so we thought trying backup might work."

Narrative can be an optional content-referenced investigator note. It is not a stable machine contract.

Capture policy is code

Every traceable field has one exact rule:

pub struct TraceCaptureRule {
    pub field_name: String,
    pub class: TraceFieldClass,
    pub mode: CaptureMode,
}

Field classes:

operational
identifier
user content
model content
evidence content
personal data
credential

Capture modes:

inline
content_reference
presence_only
omit

The policy rejects duplicate rules and unsafe combinations:

Class	Permitted modes
operational, identifier	inline, content reference, omit
user/model/evidence content, personal data	content reference, omit
credential	presence only, omit

The compiler rejects an event field with no rule, wrong class, or wrong value representation. This turns trace safety into a type and policy problem, not a best-effort regular expression over a finished log line.

Why capture-time allowlisting beats post-hoc redaction

Post-hoc redaction fails when:

the secret format is unknown;
structured data is flattened;
Unicode or encoding bypasses a pattern;
a value is split across fields/events;
a provider embeds prompts in an error;
a stack trace or debug formatter prints an entire struct;
backups or intermediate queues retain the pre-redacted copy.

Capture only permitted representations. Secret scanning remains defense in depth.

Safe inline metadata

Inline fields are limited to operational and identifier classes in the reference. Safe values are explicit variants:

SafeText(String)
Integer(i64)
Boolean(bool)

Text must be nonempty, bounded, and free of control characters. The reference additionally rejects high-signal secret shapes such as bearer tokens, common provider-key prefixes, password assignments, and private-key headers.

That detector is intentionally incomplete. It catches accidental misclassification; it is not the security boundary. The capture rule and value type are the boundary.

Per-event inline bytes are bounded. An attacker cannot place a megabyte into an “operational message” field and bypass artifact controls.

Content references

Prompts, model outputs, evidence, media, and personal data use:

ContentReference {
    content_sha256,
    bytes,
    media_type,
    access_scope_sha256,
}

The trace says which exact content was involved without copying it. The artifact store separately enforces:

tenant and principal access;
encryption;
retention and deletion;
integrity and byte bounds;
audit;
legal/purpose restrictions;
provenance and locators.

A digest is not access control

Content hashes can leak equality. Low-entropy values can be guessed and hashed. Do not hash a raw password, email address, short prompt, or API key into an ordinary trace and call it redacted.

The reference only accepts a predeclared content reference with an access-scope identity. It does not offer a hash_secret convenience method.

If equality leakage is unacceptable, use opaque random artifact IDs plus a protected integrity manifest, keyed hashes under a controlled key, or per-tenant salted identities. Document their replay implications.

Deletion and content addressing

Deleting bytes does not remove their digest from an immutable trace. Decide whether retaining the digest is permitted. A tombstone can preserve “an artifact existed” while access becomes unavailable. If even the stable identity is personal data, use an indirection strategy whose public trace token can be revoked or cryptographically erased.

Credential presence

The credential value never enters the draft type. A credential field accepts:

TraceFieldValue::SensitivePresent

and produces:

presence_only { present: true }

This can answer “was a provider credential configured?” without exposing value, length, prefix, or stable digest.

Do not record the last four characters of secrets by default. Prefixes and suffixes help attackers and can identify rotated credentials across systems. If operators need credential-version correlation, emit a separate non-secret credential record ID from the secret manager.

Redaction receipts

Each record includes:

pub struct RedactionReceipt {
    pub redaction_policy_sha256: String,
    pub source_fields: u32,
    pub inline_fields: u32,
    pub content_reference_fields: u32,
    pub presence_only_fields: u32,
    pub omitted_fields: u32,
    pub inline_bytes: u32,
    pub captured_fields_sha256: String,
}

This proves which policy and transformation class produced the stored field set. It does not prove the caller classified source data correctly. Tests, reviews, secret scanners, and constrained APIs cover that gap.

Omitted fields still create an Omitted captured marker in this compact implementation. The marker preserves schema knowledge without source value. A stricter privacy policy may omit even the field name and record only aggregate counts.

Causal structure

Wall-clock order is not causality. Event B occurring after A does not prove A influenced B.

The reference records two explicit relationships:

control parent: which earlier event enabled or scheduled this event;
data dependency: which earlier/root receipts this event consumed and produced.

run-start --start receipt--> model-dispatch
model-dispatch --dispatch receipt--> model-observed
model-observed --output receipt--> validation
validation --validation receipt--> run-terminal

The compiler requires:

the first event is parentless RunStarted;
every later event has at least one earlier parent;
every input receipt is a policy root or an earlier output;
output receipts are globally unique;
events are time ordered and inside the trace window;
the trace ends in RunTerminal;
nothing follows the terminal.

These invariants make missing provenance visible.

Multiple parents

A validation may depend on model output, evidence retrieval, and a policy decision:

model-observed ──┐
retrieval-done ──┼── validation
policy-compiled ─┘

Record all direct parents. Avoid connecting every event to every prior event; that destroys the usefulness of a causal slice.

Receipt flow

Parent edges describe control. Receipt flow describes data. Both are necessary. A scheduler may parent a tool dispatch, while the dispatch consumes an authorization receipt produced by a separate validator.

Backward causal slicing

causal_ancestors(trace, event_id) walks parent edges and returns earlier records in trace order. For run-terminal, the example returns:

run-start
model-dispatch
model-observed
validation

A production debugger should also slice by receipt dependencies:

start from the suspicious terminal or artifact receipt;
find its producer event;
collect the producer’s inputs and control parents;
recursively find their producers;
stop at declared roots;
render omitted branches explicitly.

This is a provenance slice, not proof of real-world causation. It shows declared computational dependency. A missing edge, compromised producer, or shared external cause can still mislead.

Hash chaining and integrity

Each record contains the prior record’s SHA-256 identity. The trace and record collection receive stable identities, and the terminal receipt binds:

policy;
trace;
ordered records;
event count;
emitted receipt set;
terminal record.

This detects mutation, deletion, insertion, and reordering under the stable serialization.

Hashing is not authenticity

An attacker who can rewrite the whole trace can recompute all hashes. Stronger designs anchor periodic roots in:

an append-only database with restricted writer role;
signed checkpoints;
a remote transparency log;
write-once object retention;
hardware-backed signing;
an independently operated audit sink.

Protect signing keys outside the traced process. Record key ID and algorithm, not secret key material.

Time and distributed systems

The reference uses nondecreasing integer times inside one policy window. That is enough for the deterministic fixture, not for arbitrary distributed causality.

In production record:

monotonic duration from the local process;
wall-clock timestamp and clock source;
uncertainty/skew estimate where important;
service/instance identity;
W3C trace/span context after strict validation;
sequence generated by the durable run owner;
parent/link relationships across services.

Do not sort distributed events by wall clock and infer causality. Use message/receipt IDs and span links.

Sampling

Replay and security-critical audit events should not be probabilistically absent. You may sample high-volume performance spans while retaining the run’s deterministic decision skeleton.

A practical tiering model:

Tier	Contents	Retention
required skeleton	policy, decisions, receipt joins, effects, terminal	longest justified
diagnostic metadata	safe timings, queueing, route internals	shorter
content artifacts	protected prompt/output/evidence references	purpose-specific
debug expansion	temporary high-detail safe capture	time-bounded and approved

Record sampling policy and decision. Tail sampling can retain failures after outcome is known, but buffering has memory/privacy costs.

Access, retention, and purpose

Trace access is production access. Enforce tenant, role, incident/ticket purpose, field-level authorization, and audit.

Separate:

trace metadata reader;
content artifact reader;
security auditor;
support operator;
model/evaluation engineer.

Most users who can see a route and result code do not need prompt content.

The policy’s expires_at_unix_ms bounds event admission, not storage retention by itself. Persist a retention class and deletion schedule in the storage layer. Legal holds and incident retention need explicit governance rather than silent exceptions.

Trace versus replay

A trace can say “provider returned output digest X.” Strict replay may need the exact output bytes or a content reference that still resolves. A distributed span may record timing but omit deterministic configuration. An audit record may prove authorization but not model input order.

Chapter 16 will classify replay dependencies:

required and captured;
required but intentionally unavailable;
recomputable under pinned release;
observational only;
nondeterministic external dependency.

Do not label a trace replayable until every required dependency has an explicit status.

Performance implications

Tracing consumes CPU, memory, I/O, storage, and operational attention.

Measure:

capture-rule lookup;
field validation and serialization;
digest time and bytes hashed;
event append latency;
storage bytes per run/event kind;
artifact reference cardinality;
query and backward-slice latency;
dropped/rejected trace events;
secret-scanner findings;
trace impact on request p95/p99.

The reference builds a rule map and bounded sets. A live service may compile the immutable policy once and reuse it. Do not clone large content; the draft should already contain references.

Batching writes lowers I/O but risks losing the tail on crash. Security and effect events may need synchronous durability while performance spans can buffer. Name the durability class per event.

Security implications

Log injection

Reject control characters in identifiers and inline text. Use structured encoders. Never assemble JSON with string formatting.

Cross-tenant correlation

Scope content references and trace access. Avoid globally stable digests for low-entropy personal values. Test that one tenant cannot resolve another’s artifact.

Prompt/output exfiltration

Keep content out of ordinary trace fields. Protect error paths: provider errors, validation errors, and panic/debug formatting can contain raw payloads.

Credential leakage

Use secret wrapper types without serialization or revealing Debug. This trace accepts only presence. Scan test artifacts and production sinks.

Trace forgery

Restrict append roles, fence stale workers, chain records, anchor checkpoints independently, and preserve producer release/identity.

Denial of service

Bound events, fields, parents, receipts, identifiers, inline bytes, content artifact bytes, graph queries, and retention. Reject before allocation where possible.

Malicious causal edges

Untrusted model/tool output cannot name trusted parents or receipts directly. The harness constructs edges from verified control flow and validated receipts.

Failure investigations

A secret appears in a trace export

Contain access and export. Identify the earliest unredacted copy: application draft, queue, collector, processor, sink, backup, or support download. Rotate/revoke the credential. Delete where policy and platform permit. Replace the API with a non-serializable secret type or presence/version ID. Add the exact leak shape and a structural test—not only a regex.

The trace cannot explain a terminal failure

Start at the terminal record. Walk parents and receipt inputs. Mark the first missing producer or unresolved content reference. Decide whether capture policy omitted a required replay dependency, the event append failed, or code bypassed the trace boundary. Repair the schema and add a completeness gate.

An event consumes a receipt produced later

Reject the trace. Check concurrency ownership, clock-based sorting, buffered append order, and whether receipt IDs were predicted rather than observed. Sequence by durable commit and use explicit span links for late-arriving observations.

Hash chain verifies, but records are fabricated

The writer was compromised or the whole trace was replaced. Check signed/remote anchors, writer credentials, deployment release, host evidence, and independent provider/tool audit. Hash chaining alone cannot prove authenticity.

Debug query identifies the wrong cause

Inspect whether the graph encoded temporal adjacency instead of real control/data dependencies. Look for omitted common inputs and conflated correlation IDs. Treat the slice as a hypothesis, confirm through a controlled replay or mutation experiment.

Trace storage grows without bound

Slice by event kind, field, artifact type, tenant, and debug mode. Enforce caps and retention at admission, compact only through a versioned derived summary with source identities, and preserve mandatory audit/replay skeletons.

Alternatives and trade-offs

Raw encrypted transcript

Simple for exact replay but high-risk and high-cost. Encryption does not solve overbroad access, retention, export, or application-level compromise. Use only under explicit purpose and controls.

OpenTelemetry spans only

Excellent for distributed timing/correlation. Generic attributes and sampling are not automatically a durable replay or audit format. Link an application trace identity to spans.

Event sourcing

Useful when the trace is the authoritative state history. It demands rigorous schema evolution, idempotency, snapshots, and access controls. Not every diagnostic span belongs in the domain event stream.

Database audit tables

Appropriate for authority and data mutations. Keep write paths constrained. They may omit model reasoning/input dependencies unless joined to the harness trace.

Opaque provider logs

Helpful as independent evidence but incomplete and outside your retention/access model. Import only bounded receipts and identifiers needed for reconciliation.

Runnable laboratory

The example compiles:

run-start
  -> model-dispatch
       route = safe inline identifier
       prompt = access-scoped content reference
       provider credential = presence only
  -> model-observed
       output = access-scoped content reference
  -> validation
       accepted = safe inline Boolean
  -> run-terminal

The report deliberately contains neither raw prompt bytes, raw output bytes, nor credential value. Thirteen cases cover unsafe content policy, secret-shaped inline text, missing/mismatched capture rules, parent and receipt drift, duplicate production, sequence/time/terminal violations, and field bounds.

The artifact is written to:

target/labs/harness-15/<policy-sha256>.json

Exercises

1. Classify

Choose the correct artifact and capture mode for:

model latency;
tenant ID;
raw prompt;
provider credential;
tool approval;
screenshot bytes;
validation violation code;
user email address;
final completion receipt.

2. Extend

Add a ToolAuthorized and ToolObserved branch to the reference graph. Join exact authorization, invocation, execution, and observation receipts. Make terminal validation depend on model and tool branches.

3. Redaction

Add a credential-version ID emitted by a fake secret manager. Prove it is not the credential, cannot resolve the secret, and is permitted as an inline identifier.

4. Causal slice

Implement receipt-based backward slicing in addition to parent-based slicing. Return the first unresolved root explicitly.

5. Integrity

Add signed checkpoint records every 100 events using a test key. Separate key ID/signature from the hash chain. Mutate an earlier record and prove checkpoint verification fails.

6. Retention

Design three retention classes for metadata, prompt/output artifacts, and security audit. Explain how deletion affects replay and content identities.

7. Failure investigation

Deliberately leak a fake secret through a provider error string. Find every intermediate copy, replace the unsafe type boundary, and retain scanner plus structural regressions.

8. Design

Define the minimum trace for a video generation funnel with candidate ranking, expensive-stage selection, content policy, GPU scheduling, and final artifact validation. Keep raw frames outside the trace.

Solution guidance

Metrics hold latency; identifiers may be safe inline when scoped; prompt/email/media use protected references or omission; credentials use presence/version only; approvals and completion use receipts; violation codes can be closed inline enums.
Parent edges show orchestration; receipt edges prove which exact authorization/effect/evidence flowed into validation.
Treat the version ID as a separate random non-secret identifier with its own access policy.
Index output receipts to producer events and recursively follow input receipts, stopping at policy-declared roots.
Sign a canonical checkpoint digest; do not serialize secret keys or let trace data select the trusted verification key.
Short content retention may make strict replay unavailable while metadata/audit remains. Record that state honestly.
Regex scanning catches the fixture, while the type boundary prevents recurrence.
Store frame/candidate artifacts content-addressed; trace releases, parameters, scores, decisions, reservations, and validation receipts.

Check your understanding

Why is a content digest not automatically safe redaction?
What is the difference between a control parent and an input receipt?
Why can wall-clock order not establish causality?
What does a redaction receipt prove—and not prove?
Why does hash chaining not prove authenticity?
Which events should never be probabilistically sampled away?
How can immutable trace metadata conflict with deletion requirements?
Why should untrusted output not choose parent edges?
What makes a trace sufficient for strict replay?
When is an OpenTelemetry span insufficient for application debugging?

Practical completion checklist

Authoritative references

W3C, Trace Context: interoperable distributed trace and parent identifiers plus processing requirements.
OpenTelemetry, Tracing specification: spans, links, events, sampling, status, and context propagation.
OWASP, Logging Cheat Sheet: event design, sensitive-data exclusion, injection, access, monitoring, and retention controls.
MITRE, CWE-532 · Insertion of Sensitive Information into Log File: the weakness created by logging secrets and sensitive data.
W3C, PROV-O: entities, activities, agents, derivation, and attribution vocabulary for provenance graphs.
NIST, Secure Software Development Framework 1.1: secure design, provenance, protection, review, and response practices across the software lifecycle.