Structured Task Ownership and Cancellation

Users disconnect, deadlines expire, providers stall, deployments shut down, and policies change. Continuing to spend model/GPU/tool resources after a result is unwanted is a correctness bug.

Structured task ownership means a parent knows its children and does not declare completion while they remain unobserved.

Learning objectives

• own every spawned child through a handle/supervisor;
• distinguish dropping futures, aborting tasks, and cooperative cancellation;
• propagate cancellation through a cloneable token;
• use select! while defining cleanup and race semantics;
• join children and classify panics/cancellation;
• design cancellation-safe steps and external-effect receipts;
• test cancellation deterministically;
• run the companion cancellation example.

Dependency and mental model

This chapter depends on futures/tasks and ownership. It precedes channels/backpressure, durable jobs, SSE disconnects, and graceful shutdown.

parent run
├── evidence fetch
├── provider call
└── tool child
    cancellation flows downward
    outcomes/errors flow upward
parent terminal event only after owned children settle

Cancellation mechanisms

Drop a future

When a future is dropped, it is no longer polled. This is immediate locally but may leave remote effects/work.

Abort a spawned task

JoinHandle::abort requests task cancellation at the next await/yield boundary. Destructors of live state run, but synchronous code cannot be interrupted safely mid-instruction.

Cooperative token

The companion Cancellation uses a Tokio watch channel:

tokio::select! {
    result = operation => result,
    _ = cancellation.cancelled() => Err(RunError::Cancelled),
}

Children periodically await/select the token and perform explicit cleanup.

Use cooperative cancellation for semantic cleanup. Retain abort as a bounded shutdown fallback.

The companion token

#[derive(Clone)]
pub struct Cancellation {
    sender: watch::Sender<bool>,
}

cancel stores true; new subscribers observe already-issued cancellation. This avoids a lost one-shot notification:

pub async fn cancelled(&self) {
    let mut receiver = self.sender.subscribe();
    if *receiver.borrow_and_update() {
        return;
    }
    // wait for change...
}

Cancellation is idempotent. It carries no reason yet; production tokens may pair a stable reason with the signal in run state/trace.

select! races need policy

If completion and cancellation become ready together, branch selection may vary unless biased/order behavior is specified. Decide whether:

• completed durable result wins;
• cancellation wins before committing;
• operation commit is reconciled by state/version;
• terminal event is derived from persisted truth.

Do not infer “the cancellation branch ran” means the provider/tool had no effect.

For side effects:

1. assign idempotency key;
2. record intent;
3. invoke;
4. record receipt/unknown outcome;
5. reconcile after cancellation/crash.

Join every child

let child = tokio::spawn(work(token.clone()));
// ...
token.cancel();
match child.await {
    Ok(outcome) => { /* observed */ }
    Err(join_error) if join_error.is_cancelled() => { /* aborted */ }
    Err(join_error) if join_error.is_panic() => { /* defect */ }
    Err(join_error) => { /* classify */ }
}

A detached task can outlive tenant/request authorization, retain large Arcs, and emit events after terminal status.

For multiple children use a task set/supervisor that:

• assigns child IDs;
• cancels siblings under policy;
• drains every join result;
• bounds child count;
• records panic/cancellation;
• has shutdown deadline.

Cancellation-safe state transitions

Avoid marking a run completed before trace/output durability:

generate → verify → persist artifact → append completion → publish terminal

If cancellation happens between persist and terminal event, recovery must discover the artifact and finish/reconcile idempotently.

Cleanup should itself have a deadline. A stuck cleanup cannot block deployment forever. Escalate from cooperative cancellation to abort/process kill while recording incomplete cleanup.

Provider/tool/media specifics

• HTTP request drop may close the connection but provider compute/billing can continue.
• Subprocess cancellation must signal, wait, then force kill; drain pipes to avoid deadlock.
• GPU kernels may continue after host task cancellation.
• spawn_blocking work generally cannot be forcibly stopped once running; use cooperative flags or isolate in a process.
• database transaction/drop semantics depend on driver; test rollback.
• partial files need temp names and atomic publish after verification.

Deterministic cancellation tests

Do not use arbitrary sleeps. Synchronize the test:

1. child signals “started” through oneshot;
2. parent sends cancellation;
3. child observes token and returns;
4. parent awaits join;
5. assert no later events/effects.

The example and test follow this pattern.

Runnable companion implementation

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example structured_cancellation
cargo test -p mosaic-runtime --examples

Expected:

worker: cancelled

examples/structured_cancellation.rs owns the JoinHandle, waits for start, cancels, and joins.

Failure investigation: run is cancelled but model calls continue

Check:

1. cancellation reaches the provider future/task;
2. spawned handles are retained;
3. SDK request is cancellation-safe;
4. retries stop after cancellation;
5. remote cancellation API exists;
6. billing/receipts show post-cancel work;
7. terminal event occurs after child drainage.

If remote compute cannot cancel, present cancellation as “result no longer awaited” and account for possible cost honestly.

Security and performance implications

Security:

• cancellation must not bypass audit/receipt persistence;
• authorization revocation may require cancelling pending tools;
• clean partial sensitive artifacts;
• cap cleanup time;
• process-isolate unsafe native work when abort is insufficient.

Performance:

• cooperative checks have low cost;
• too-frequent polling can add overhead;
• cancellation storms can overload cleanup/storage;
• detached tasks create hidden resource tails;
• join supervision consumes bounded state per child.

Exercises

1. Add cancellation reasons and terminal trace event.
2. Supervise three children; cancel siblings on required-child failure.
3. Add cleanup deadline then abort.
4. Test cancellation before start, during wait, after durable commit.
5. Design subprocess cancellation with SIGTERM/kill equivalents.

Solution guidance

Store reason in run state, not only the broadcast token. Drain all join results even after one failure. Use synchronization points, not sleep. Treat post-commit cancellation as reconciliation, not rollback fiction.

Check your understanding

• Does dropping a JoinHandle cancel its task?
• Can spawn_blocking be forcibly interrupted safely?
• What if completion and cancellation are simultaneously ready?
• Why join after cancel?
• How do idempotency receipts help an unknown outcome?

Completion checklist

• Every child has an owner.
• Cancellation flows down and outcomes flow up.
• Cleanup and shutdown are bounded.
• External effects use idempotency/receipts.
• Tests synchronize deterministically.
• I ran the cancellation example/test.

Authoritative references

• tokio::select!
• tokio::task::JoinHandle
• Tokio graceful shutdown
• Tokio task cancellation safety