Day 31 — Streaming, Backpressure, and Cancellation

Today Mosaic handles work that is larger and slower than one HTTP request.

Bound every queue

An unbounded channel is a delayed out-of-memory failure. Define capacity and the behavior when full: reject admission, wait with a deadline, shed low-priority work, or persist for later.

let (tx, rx) = tokio::sync::mpsc::channel::<Job>(capacity);
let permits = Arc::new(tokio::sync::Semaphore::new(max_inference));

Admission, queue capacity, and inference permits solve different problems. Measure queue time separately from execution time.

Stream with resumption

Use ordered event IDs for server-sent events. A reconnect supplies the last received ID and resumes from durable events. Do not rerun inference merely because the client connection dropped.

Uploads stream to bounded temporary or object storage while hashing and enforcing maximum bytes. Never buffer an untrusted video fully in request memory.

Propagate cancellation

Client disconnect may cancel presentation but not an external effect already dispatched. Structured tasks need an owner, cancellation token, deadline, and join on shutdown. Indeterminate effects enter reconciliation.

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example bounded_backpressure
cargo run -q -p mosaic-runtime --example blocking_work

Overload drill

Simulate slow consumers and more jobs than capacity. Prove bounded memory, stable rejection, preserved event order, no orphan tasks, and useful retry guidance.

Completion proof

Document each buffer, queue, semaphore, and timeout in the request path. If one has no numeric bound or owner, the system is not production-ready.

Deep references: Channels, semaphores, and backpressure, Streaming uploads and SSE, and Structured tasks and cancellation.

Next: Day 32 — Security and Tenancy →.