Futures, Executors, and Tokio's Runtime Model

AI applications wait on provider HTTP calls, database queries, object storage, subprocesses, and streams. Rust async code lets a small set of runtime threads coordinate many waiting operations.

Async does not make CPU work faster, create capacity, or automatically run operations in parallel. It changes how waiting work yields control.

Learning objectives

After this chapter, you will be able to:

• explain a future as a lazy state machine polled toward completion;
• distinguish a future, task, executor, runtime, thread, and blocking operation;
• describe what .await may do to control flow and borrows;
• choose sequential awaits, join!, or spawned tasks from ownership needs;
• configure Tokio features/runtime ownership deliberately;
• avoid nested runtimes and CPU blocking on worker threads;
• reason about task scheduling without assuming fairness;
• run the companion’s concurrent I/O example.

Dependencies and downstream use

This chapter depends on ownership, moves, closures, Send, and errors. It precedes structured cancellation, bounded channels, blocking work, HTTP, and durable workers.

async fn call()
    │ invocation creates a Future (no body progress yet)
    ▼
executor polls Future
    ├── Ready(output)  → complete
    └── Pending + Waker
             │ resource becomes ready
             └── task scheduled for another poll

Futures are lazy state machines

Conceptually:

trait Future {
    type Output;
    fn poll(
        self: Pin<&mut Self>,
        context: &mut Context<'_>,
    ) -> Poll<Self::Output>;
}

Calling an async fn constructs a future. Work advances when an executor polls it. When an awaited resource is not ready, the future returns Pending after arranging a wake-up.

Do not manually implement Future for ordinary application work. Understanding poll explains:

• why blocking a poll blocks the worker;
• why futures are lazy;
• why cancellation by drop can stop future progress;
• why pinning matters for self-referential compiler-generated state;
• why a resource must wake the task when it becomes ready.

.await is a suspension point

let response = provider.infer(request).await?;

If ready, execution continues immediately. If pending, the task yields so the runtime thread can poll other tasks. Locals needed later remain stored in the future state machine.

Review every await for:

• owned/borrowed values held across suspension;
• lock/semaphore/file resources held;
• cancellation safety;
• deadlines;
• whether the awaited operation is actually nonblocking.

An async function can still execute long synchronous work before its first await or between awaits.

Executor, task, and runtime

• A future is the computation/state machine.
• A task is a future scheduled independently by an executor.
• An executor polls ready tasks.
• Tokio’s runtime includes an executor, I/O driver, timer driver, blocking pool, and configuration.
• An OS thread executes runtime polling or blocking work.

Tokio’s multi-thread runtime may move Send tasks among worker threads. A task is not permanently assigned one thread. Thread-local/native-thread-confined state therefore needs a different architecture.

#[tokio::main] builds a runtime and blocks the process entry thread until the async main future finishes. Libraries should normally expose async functions and let the application own runtime creation.

Sequential versus concurrent composition

Sequential:

let evidence = fetch_evidence().await?;
let policy = fetch_policy().await?;

Use when the second depends on the first or concurrency would waste/violate capacity.

Concurrent in one parent task:

let (evidence, policy) = tokio::join!(
    fetch_evidence(),
    fetch_policy(),
);

join! polls both futures but does not spawn new tasks or bypass the current runtime thread. It is suitable for independent async I/O with structured parent ownership.

For fallible futures, decide whether one error should cancel/drop siblings and whether cleanup is safe. The try_join! family short-circuits on error.

Spawning changes ownership

let run_id = run_id.clone();
let handle = tokio::spawn(async move {
    process(run_id).await
});

Spawning:

• requires independently owned/captured state (commonly 'static);
• usually requires Send on the multi-thread runtime;
• returns a JoinHandle;
• separates child scheduling/lifetime unless the parent retains and awaits the handle.

Do not spawn merely to “make it async.” Use spawning for real independent task ownership or parallel/concurrent supervision. The next chapter ensures children are cancelled and joined.

Dropping a JoinHandle detaches the task by default; it does not necessarily cancel the work.

Runtime configuration

The companion enables only required Tokio features:

tokio = {
  version = "1",
  features = [
    "fs", "macros", "rt-multi-thread",
    "signal", "sync", "time"
  ]
}

Avoid features = ["full"] without need; feature choices affect dependency/build/attack surface.

Runtime decisions include:

• current-thread versus multi-thread scheduler;
• worker thread count;
• blocking thread limit;
• thread names/hooks;
• timer/I/O enablement;
• shutdown timeout.

Defaults are starting points. Measure task latency, queueing, blocking, CPU saturation, and runtime metrics under representative loads.

Do not nest/block runtimes casually

Calling Runtime::block_on inside an async Tokio task can panic or deadlock depending on context. Likewise, synchronous library APIs that secretly create runtimes make composition difficult.

Boundary rule:

• process/CLI/test owns the runtime;
• async library exposes async;
• synchronous CPU helper stays synchronous;
• explicit adapter bridges sync/async at one reviewed edge.

For tests, use #[tokio::test] or a shared explicit test runtime. Do not create one runtime per small operation in production.

Cooperative scheduling

Tokio tasks cooperate by returning Pending/awaiting. A long CPU loop without awaits can monopolize a worker:

async fn bad_decode(bytes: &[u8]) {
    expensive_decode(bytes); // synchronous CPU work on runtime worker
}

An inserted yield_now is rarely the right fix for heavy media/model computation. Move blocking/CPU work to a bounded blocking/dedicated pool, process, or service. Chapter 4 implements this.

Do not assume scheduler fairness, polling order, or deterministic completion. External artifacts must sort by stable IDs/timestamps where order matters.

Async I/O is only async when the implementation is

Tokio file APIs may use blocking-pool operations for ordinary files. Some SDK methods named async can perform synchronous parsing/compression. Native inference APIs may block the calling thread.

Classify work by measurement and implementation:

• socket readiness: async I/O;
• provider client plus large synchronous JSON transform: mixed;
• image/video decode: CPU/blocking/native;
• GPU launch: may return before device completion;
• SQLite call: depends on driver/runtime design.

Type syntax alone does not prove nonblocking behavior.

Cancellation by drop

Dropped futures stop being polled. Whether this safely cancels an operation depends on the future:

• socket read usually releases registration/resources;
• database transaction wrapper may roll back on drop;
• remote provider may continue server-side;
• subprocess/native call may continue;
• side effect may have happened before the await returns.

Every async capability should document cancellation safety and external effect semantics. The next chapter adds explicit cooperative cancellation and child joining.

Runnable companion implementation

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example futures_runtime

Expected output:

joined: evidence + policy

crates/mosaic-runtime/examples/futures_runtime.rs starts two timer-backed futures with different delays and joins them in one parent task. Output remains argument order, not completion order.

The example is intentionally I/O-shaped. CPU work appears in the blocking chapter.

Failure investigation: provider traffic is slow despite low CPU

1. inspect whether calls are awaited sequentially by accident;
2. verify they are independent and concurrency is allowed;
3. inspect connection pool/DNS/TLS/provider limits;
4. measure queue and per-stage latency;
5. use join! or bounded spawned work;
6. preserve a global/provider/tenant capacity limit;
7. compare tail latency, cost, and error rate.

Unbounded join_all over user-sized input is not the fix.

Common failure modes

• CPU decode directly in async task.
• tokio::spawn with discarded/detached handle.
• assuming async fn starts immediately.
• holding locks/permits across unrelated awaits.
• nested runtime creation in libraries.
• unbounded concurrent futures.
• assuming completion order is stable.

Security and performance implications

Security:

• bound concurrent/queued work before spawning;
• cancel on authorization/policy revocation when required;
• do not keep secrets in long-lived detached tasks;
• use deadlines and input limits before expensive async work;
• trace runtime task ownership without raw sensitive payloads.

Performance:

• async excels at waiting concurrency, not CPU acceleration;
• each task/future retains its live local state;
• spawning millions of small tasks consumes memory/scheduling;
• excessive boxing/dynamic futures can matter at very high call rates;
• provider inference usually dominates one future allocation.

Exercises

Recall

1. When does an async function body begin advancing?
2. What wakes a pending future?
3. Does join! spawn tasks?
4. What happens when a JoinHandle is dropped?

Implementation

1. Compare sequential awaits with join! using deterministic delays.
2. Add typed failure to one branch and test try_join!.
3. Record start/completion IDs and prove artifact sorting is stable.
4. Configure a current-thread runtime for a test and explain constraints.

Design

Classify provider calls, SQLite, image decode, FFmpeg, tokenization, GPU inference, SSE, and hashing. Choose async, blocking pool, dedicated worker, or external service and justify ownership/capacity.

Solution guidance

Measure elapsed time only as a demonstration, not a robust benchmark. try_join! drops sibling futures on first error, so verify cancellation safety. Stable output should sort by IDs rather than timer completion.

Check your understanding

• Can an async function block a Tokio worker?
• Why might a future need pinning?
• Does spawning make CPU work parallel safely?
• Who should own the runtime?
• Is dropping a future equivalent to undoing an external effect?

Completion checklist

• I distinguish future/task/executor/runtime/thread.
• I know every suspension point and held resource.
• I compose concurrency only under explicit capacity.
• Libraries do not secretly create nested runtimes.
• CPU/native work is not mislabeled as async I/O.
• I ran the joined-futures example.

Authoritative references

• The Rust Book: Asynchronous Programming
• std::future::Future
• Tokio: Async in depth
• Tokio runtime
• tokio::join!
• tokio::spawn