Unit, Integration, Golden, Property, Fuzz, and Chaos Tests

An AI application has two kinds of uncertainty:

software uncertainty: races, corrupt state, boundary bugs, retries, crashes, malformed bytes;
model uncertainty: variable outputs, capability limits, provider change, and distribution shift.

Do not mix them. A live model call is not a substitute for a unit test, and a unit test cannot prove model quality. First make the deterministic shell trustworthy; then evaluate the probabilistic component with versioned datasets and statistical gates.

The companion mosaic-testing crate makes this executable. It supplies a manual clock, semantic failpoints, an end-to-end lease-expiry recovery drill, a reviewed golden report, three property tests, checked-in fuzz seeds, and a separately compiled cargo-fuzz target.

Learning objectives

choose the smallest test type that can falsify a specific claim;
design pure unit tests and public-boundary integration tests;
use golden files as reviewed contracts rather than automatic snapshots;
express mathematical and stateful invariants with property tests;
build bounded raw-byte and structure-aware fuzz targets;
convert every fuzzer artifact into a deterministic regression;
inject crashes at semantic boundaries with controllable time;
prove lease recovery, fencing, idempotency, and terminal uniqueness;
understand concurrency-model testing limits;
separate deterministic CI, fuzzing, chaos, and live-model evaluation.

Dependencies and mental model

This chapter depends on valid domain types, durable jobs, lease generations, idempotency, transactional events, quotas, and cancellation.

Testing is an attempt to falsify claims:

claim
  ├── one known example?                    → unit/example test
  ├── public crates + real adapter?          → integration test
  ├── exact reviewed representation?         → golden test
  ├── for all generated values/sequences?    → property/state-machine test
  ├── hostile bytes over time?               → fuzz test
  ├── interleavings?                         → model/concurrency test
  ├── process/network/storage failure?        → fault/chaos test
  └── model behavior on a distribution?       → evaluation

More “realistic” is not automatically better. A test with an uncontrolled clock, external model, shared database, network, and random retry may be realistic but unable to reproduce its own failure.

Start with behavior and invariants

Write the contract before the test:

accepted work is either durably queued or explicitly rejected;
one idempotency key and fingerprint identify one run;
event sequence is strictly increasing;
only the current unexpired lease generation may commit;
a terminal transition happens at most once;
quota active/reserved counts never become negative;
every decoded task can be serialized and decoded again;
cancellation or failure releases owned capacity.

Then list the adversary for each invariant: duplicate request, stale worker, clock boundary, integer overflow, malformed JSON, Unicode, concurrent admission, partial write, retry, or process death.

A coverage percentage does not state any of these claims.

Layer 1: unit tests for local decisions

Unit tests sit beside the implementation and can observe private helpers. Use them for:

budget and quota arithmetic;
parser/domain distinction;
state transition tables;
retry schedule calculation;
selector tie-breaking;
redaction;
deterministic normalization.

The companion manual clock test has no sleep:

let clock = ManualClock::new(40)?;
let clone = clock.clone();

assert_eq!(clone.now()?, 40);
assert_eq!(clock.advance(2)?, 42);
assert_eq!(clone.now()?, 42);

The clock rejects negative values and checked-add overflow. Its important property is not that it resembles a platform clock; it exposes time as an input. Production adapters obtain real time once at their boundary and pass the timestamp into policy/storage methods.

Test errors by stable variants/codes, not complete human prose. Error wording is documentation; machine behavior should not depend on punctuation.

Layer 2: integration tests through public APIs

Cargo treats files under tests/ as separate crates. That means an integration test cannot reach private implementation accidentally. It exercises the API a consumer actually receives.

The companion recovery test uses:

public mosaic-db APIs;
real embedded SQL migrations;
an isolated SQLite database;
persisted runs, jobs, idempotency keys, and events;
explicit time and workers.

The acceptance case:

create run/job/idempotency/event atomically
→ worker A claims generation 1
→ A dies before completion
→ lease reaches expiry
→ worker B claims generation 2
→ A's late completion returns LostLease
→ B commits exactly one terminal event
→ another completion returns LostLease

This is not a mocked repository test. It crosses domain, SQLx, migration, transaction, and decoding boundaries. It remains deterministic because it uses a local database and manual time.

Use an external service only when its behavior is the contract under test. For example, test PostgreSQL locking against the supported PostgreSQL version in a dedicated job; do not pretend SQLite proves SKIP LOCKED behavior.

Layer 3: golden contracts

A golden file freezes an exact representation that humans care about:

CLI output;
event/trace schema;
normalized evidence manifest;
edit decision list;
generated API schema;
deterministic report.

The recovery test serializes RecoveryReport and compares it to crates/mosaic-testing/fixtures/recovery-report.golden.json:

{
  "run_id": "run-recovery-contract",
  "first_generation": 1,
  "recovered_generation": 2,
  "stale_worker_fenced": true,
  "final_status": "completed",
  "event_kinds": [
    "queued",
    "completed"
  ]
}

The file makes generation fencing and terminal history reviewable. If a field/order changes, the test fails. The correct response is:

inspect the semantic diff;
decide backward compatibility and migration;
change implementation or intentionally review the new golden;
record why the contract changed.

Never have CI auto-accept snapshots. Remove timestamps, random IDs, unstable map order, host paths, and secrets before snapshotting. If instability is meaningful, represent it as a typed range or separate assertion instead of erasing it.

Layer 4: property tests

An example proves one point. A property test generates many points and shrinks failure toward a small counterexample.

Derive the property, then derive the generator

The domain budget invariant is:

valid ⇔ max_model_calls > 0 ∧ max_repairs < max_model_calls

The companion generates all three integer fields and checks the implementation against that independent expression:

proptest! {
    #[test]
    fn budget_validation_matches_its_mathematical_invariant(
        calls in any::<u32>(),
        repairs in any::<u32>(),
        bytes in any::<usize>(),
    ) {
        let budget = RunBudget {
            max_model_calls: calls,
            max_repairs: repairs,
            max_output_bytes: bytes,
        };
        prop_assert_eq!(
            budget.validate().is_ok(),
            calls > 0 && repairs < calls
        );
    }
}

If the generator only creates already-valid values, it cannot test rejection. If it generates mostly nonsense that fails in the first parser branch, it cannot explore deep behavior. Shape the distribution around boundaries: zero/one, exact limit/limit plus one, empty/one/maximum length, expiry minus one/exact expiry/plus one.

Stateful/resource properties

The quota properties generate reservation sequences and assert:

dropping any admitted reservations restores active_runs = 0;
dropping restores reserved_units = 0;
no units are charged on drop;
committing bounded actual usage leaves no reservation;
charged usage equals the independent sum of actual values.

The implementation is not used to calculate the expected sum. Otherwise the test can repeat the same defect in its oracle.

Useful AI-system properties:

evidence locators remain within source bounds after normalization;
text chunk union preserves the intended source span;
image rectangles remain within decoded dimensions;
audio/video timestamps remain monotonic after timebase conversion;
token allocation never exceeds run budget;
serialization round-trips;
authorized capability sets never gain privileges under projection;
retry count never exceeds the budget;
terminal states reject further transitions.

Persisted Proptest failures must be kept or promoted so CI replays them. Control case counts and shrinking time by risk; “10,000 cases” is not proof over an infinite space.

Layer 5: fuzz hostile boundaries

Property testing starts from structured values selected by a strategy. Coverage-guided fuzzing mutates inputs and retains those that reach new program paths. Raw-byte fuzzing is appropriate for:

JSON/protocol decoders;
image/audio/video headers;
archive/container parsing;
UTF-8 and path boundaries;
model-produced structured output repair;
trace/event readers;
FFI wrappers.

The independent fuzz workspace contains fuzz_targets/task_json.rs:

fuzz_target!(|bytes: &[u8]| {
    if bytes.len() > 64 * 1024 {
        return;
    }
    if let Ok(task) = serde_json::from_slice::<Task>(bytes) {
        let _validation = task.validate();
        let encoded = serde_json::to_vec(&task)
            .expect("decoded tasks always serialize");
        let decoded: Task = serde_json::from_slice(&encoded)
            .expect("serialized tasks always decode");
        assert_eq!(task, decoded);
    }
});

The size guard is part of the security contract. The target must not perform external network calls, write arbitrary paths, or hide panics. The checked-in corpus includes valid, invalid-domain, malformed, and multilingual/path-shaped task inputs. An ordinary stable-toolchain test reads these same seeds so they never silently rot.

Run:

cd rust/rust-ai-engineering
cargo check --manifest-path fuzz/Cargo.toml
cargo +nightly fuzz run task_json -- -max_len=65536 -max_total_time=60

cargo-fuzz/libFuzzer needs supported sanitizer infrastructure and a nightly toolchain. A compile check proves the target builds; only a timed/continuous fuzz campaign explores it.

Artifact workflow

For every crash/hang/OOM:

preserve exact target, commit, options, sanitizer, and artifact;
reproduce the artifact;
minimize it;
classify panic, memory error, unbounded resource use, or false oracle;
fix the smallest violated invariant;
promote minimized input to an ordinary regression test/corpus;
rerun relevant properties and fuzz campaign.

Do not merely delete fuzz/artifacts.

Structure-aware fuzzing

Raw mutations may spend all time failing syntax. Structure-aware targets can generate:

sequences of run state transitions;
edit decision lists;
evidence graph nodes/edges;
media metadata with valid containers but adversarial timing;
tool call/result sequences.

This explores deeper semantics but can miss malformed syntax. Keep both raw and structured targets.

Layer 6: deterministic fault and chaos tests

Randomly killing processes can find failures but makes causal debugging expensive. Start with semantic failpoints:

pub enum FaultPoint {
    AfterClaim,
    BeforeCompletion,
    BeforeRetry,
}

ScriptedFaults schedules an exact (point, hit_number). It counts under a mutex and returns a typed InjectedFault. The recovery example fails on the first BeforeCompletion, not “roughly 5% of the time.”

Semantic boundaries matter:

before/after durable admission commit;
after provider charge but before receipt persistence;
after artifact bytes but before metadata publish;
before/after tool side effect and idempotency receipt;
after claim/lease renewal;
before terminal event/projection commit;
during stream reconnect/cursor replay.

The companion drill advances to exact lease expiry because the SQL contract uses lease_expires_at <= now. Worker B receives generation 2. Worker A’s generation-1 completion is rejected even if it wakes later. This proves the safety mechanism, not just eventual success.

After deterministic points pass, expand:

process/container kill at each phase;
database connection reset/transaction abort;
object-store timeout, partial read, checksum mismatch;
provider timeout/429/5xx after uncertain billing;
duplicate/delayed/reordered messages;
full disk and read-only filesystem;
clock skew at external protocol boundaries;
CPU/GPU OOM and cancellation;
dependency outage and recovery storm.

Chaos has a hypothesis, blast-radius bound, stop condition, and invariant monitor. “Break random things in production” is not an engineering method.

Concurrency interleavings

Stress tests can run racing operations repeatedly, but a pass means only that sampled schedules passed. Loom can explore permutations only for synchronization operations replaced with Loom’s types. Hidden std/Tokio synchronization remains invisible, and state spaces grow rapidly.

Use:

ordinary synchronized concurrency tests for public behavior;
Loom for small concurrency primitives designed for model checking;
database integration tests for transaction/isolation behavior;
failpoints for distributed/process boundaries;
load tests for scheduler/resource behavior.

Do not claim an exhaustive concurrency proof after setting a preemption/permutation bound. Record the bound.

Tests versus model evaluations

Deterministic software CI uses scripted/recorded model adapters. Live model evaluation is separate:

software gate
  exact code + fixtures + no provider variability

model evaluation
  model/provider version + harness version + dataset split
  + sampling policy + cost ceiling + statistical comparison

A small smoke evaluation may run after deterministic gates, but it must not make parser, transaction, or cancellation regressions intermittent. Never overwrite recorded fixtures from a live provider during CI.

Test data and isolation

Fixtures are products:

schema/version and origin;
rights/consent;
non-secret synthetic values where possible;
separate software, development-eval, held-out, safety, and red-team sets;
stable IDs;
no developer machine paths;
cleanup ownership;
minimized reproduction attached to defects.

Each test owns an isolated database/path/port/tenant. Avoid wall-clock uniqueness alone; parallel tests can collide. Use OS-provided temporary directories or random plus process/atomic identity, and retain failed fixtures only under explicit safe policy.

Tests must not require an execution order. If a test mutates global environment variables, process working directory, tracing subscriber, or static state, serialize/isolate that boundary or refactor it into injected state.

Runnable companion implementation

cd rust/rust-ai-engineering
cargo test -p mosaic-testing --all-features
cargo clippy -p mosaic-testing --all-targets --all-features -- -D warnings
cargo run -q -p mosaic-testing --example recovery_drill
cargo check --manifest-path fuzz/Cargo.toml

The deterministic suite currently contains:

three unit tests for manual time and exact failpoint schedules;
one corpus regression covering accept/decode/domain-reject/malformed paths;
three Proptest properties;
one end-to-end recovery/invariant test;
one reviewed golden test;
one separately compiled libFuzzer target.

Expected example:

{
  "run_id": "run-recovery-demo",
  "first_generation": 1,
  "recovered_generation": 2,
  "stale_worker_fenced": true,
  "final_status": "completed",
  "event_kinds": ["queued", "completed"]
}

Failure investigation: a test flakes only in CI

preserve seed, order, worker count, platform, toolchain, environment, and full output;
rerun the exact test alone and with the suite/repetition;
inspect wall clocks/sleeps, shared paths/ports/environment/global state;
inspect map order, generated IDs, locale/timezone, and thread scheduling assumptions;
check leaked tasks/resources and missing joins;
replace time/random/external dependencies with controlled inputs;
add a regression that fails under forced adverse order.

Increasing timeout or retrying a failing test hides evidence unless timeout is the proven contract.

Failure investigation: a fuzz target finds OOM, not panic

preserve and minimize input with resource limits active;
identify allocation/loop/decompression expansion before validation;
ensure transport and parser limits agree;
parse metadata incrementally and checked-arithmetic dimensions/counts;
bound nesting, collection length, output, recursion, and decompressed size;
add exact input as deterministic regression;
rerun under sanitizer and longer campaign.

Rust memory safety does not prevent deliberate memory exhaustion.

Failure investigation: two workers completed one run

preserve job rows, generation/owner/expiry, run projection, events, and receipts;
determine whether completion checked current generation inside the terminal transaction;
inspect lease expiry comparison and clock source;
inspect external side effect idempotency separately from database terminal uniqueness;
quarantine duplicate downstream effects;
add deterministic stale-worker and exact-expiry tests;
add database constraints/conditional update before relying on workers.

Alternatives and trade-offs

Handwritten table tests

Readable, stable, excellent for named boundaries. They sample only enumerated cases.

Property testing

Explores broad structured input and shrinks. Properties/generators can be wrong, and case count is not a proof.

Golden files

Make exact change review easy. They become noisy if representation includes nondeterminism or if reviewers accept updates blindly.

Coverage-guided fuzzing

Excellent for hostile parser/protocol surfaces over time. Requires toolchain/sanitizer resources and a corpus/artifact lifecycle.

Deterministic failpoints

Reproducible and causally precise. They cover only instrumented semantic boundaries.

External chaos

Tests real deployment behavior. It costs more, has blast radius, and needs invariant monitoring and cleanup.

Security and performance implications

Security:

fuzz every untrusted byte boundary and parser/FFI wrapper;
never put production credentials or personal media in fixtures/corpora/artifacts;
sandbox fuzz/chaos targets and bound input/time/memory/disk;
test cross-tenant and missing-capability paths, not only success;
redact failure logs and golden diffs;
treat test dependencies, CI tokens, and artifacts as supply-chain surfaces.

Performance:

keep fast deterministic unit/property smoke cases in the main loop;
shard integration tests by real bottleneck, not arbitrary file count;
use fixed property seeds to reproduce, then vary seeds in broader jobs;
keep golden content minimal and semantic;
fuzz continuously/time-bounded outside the normal fast gate;
measure chaos recovery time without replacing correctness assertions;
do not share one globally contended fixture unless contention is under test.

Exercises

Add a property that every valid TenantId survives Serde round-trip and rejected strings cannot enter through deserialization.
Add a raw-byte fuzz target for JSONL trace reading without arbitrary filesystem writes.
Add AfterArtifactBytes/BeforeMetadataPublish failpoints and prove no corrupt artifact becomes visible.
Run the recovery drill at expiry minus one, exact expiry, and plus one.
Model a two-worker reservation primitive with Loom and document the explored bound/limitations.
Add a PostgreSQL integration job proving concurrent claim behavior under the supported version.
Design a chaos experiment for provider success followed by lost response; specify idempotency and billing reconciliation.

Solution guidance

Generate identifiers from small Unicode/control/length classes, and compare constructor acceptance with Serde acceptance. Extract trace decoding to a byte/reader API so the fuzz target does not need a path. Artifact publication should use a hidden temporary object/file, verified digest, and conditional atomic publish; failpoint tests assert readers never observe the temporary name. At minus one, the second claim is absent; at exact/plus one, it is generation 2 and generation 1 is fenced. A Loom model must use Loom replacements for every relevant synchronization operation. PostgreSQL tests need isolated schema/database and synchronized racing claims. Provider uncertainty needs a stable operation key, receipt lookup/reconciliation, and no blind duplicate charge.

Check your understanding

Why is a live model call a poor software unit-test dependency?
What does an integration test prove that a repository mock cannot?
When is a golden change correct?
Why generate invalid values in a property strategy?
What is shrinking?
Why keep raw and structure-aware fuzz targets?
What must happen to a minimized fuzz artifact?
Why is a manual clock stronger than a short sleep?
What does lease generation fence?
Why is a passing stress test not an exhaustive concurrency proof?