Day 30 — Durable Jobs and Artifacts

Today Mosaic survives process death.

Run the durability examples

cd rust/rust-ai-engineering
cargo run -q -p mosaic-db --example transactional_run
cargo run -q -p mosaic-db --example durable_job
cargo run -q -p mosaic-artifacts --example content_addressed

Commit state and event together

When a run changes state, persist the new state and corresponding event in one transaction. Otherwise a crash can leave current state saying completed while replay has no terminal event—or the reverse.

Lease, do not merely dequeue

A worker claims a job until lease_expires_at, periodically renews, and completes with an ownership token. If it dies, another worker may reclaim after expiry. Completion from a stale lease is rejected.

Idempotency keys bind logical operations to outcomes. They do not make every external API idempotent; for uncertain effects, reconcile target state before retry.

Store large data outside rows

Content-addressed artifacts use a digest as identity and store metadata in the database. Verify bytes on read, scope access by tenant, and retain relationships between original, normalized, derived, and model output artifacts.

Crash drill

Inject failure:

1. after job claim;
2. after provider result but before event commit;
3. after artifact write but before database link;
4. during lease renewal;
5. after external write returns an unknown outcome.

For each, state the recovery owner and whether replay, retry, garbage collection, or reconciliation is correct.

Completion proof

Run the same logical request twice with one idempotency key, restart between attempts, and prove one durable run and one externally visible outcome.

Deep references: Durable jobs, SQLx transactions, and Databases and content addressing.

Next: Day 31 — Streaming and Backpressure →.