Blocking CPU and Media Work, Accelerator Scheduling

Image decode, audio resampling, video demux, hashing, tokenization, native inference, and FFmpeg may block or consume sustained CPU. GPU/NPU work adds device memory and asynchronous completion. These jobs need separate scheduling from socket/database waits.

Learning objectives

• classify CPU, blocking syscall, subprocess, and accelerator work;
• use spawn_blocking only with explicit concurrency bounds;
• understand its cancellation/shutdown limitations;
• choose blocking pool, dedicated Rayon/pool, one-owner thread, subprocess, or service;
• schedule by memory/device/tenant cost, not task count alone;
• prevent CPU/GPU oversubscription;
• benchmark queue plus execution;
• run the bounded blocking companion example.

Dependency and model

Depends on runtime, cancellation, semaphores.

Tokio async workers: readiness + short polls
        │ submit bounded
        ▼
blocking/CPU pool: decode/hash/tokenize
        │ bounded artifact
        ▼
accelerator scheduler: memory/residency/batch

spawn_blocking

let result = tokio::task::spawn_blocking(move || decode(input))
    .await
    .map_err(DecodeError::Join)??;

It moves synchronous work to Tokio’s blocking pool. The pool can grow many threads; it is not a capacity policy for CPU-heavy jobs. Pair with a semaphore or use a dedicated fixed pool.

Once blocking work starts, aborting the async wrapper generally cannot stop the closure. Runtime shutdown may wait for it. Long/hostile native decoders may require cooperative flags or subprocess isolation.

Companion BoundedExecutor

pub async fn run_blocking<F, T>(&self, operation: F)
    -> Result<T, ExecutionError>
where
    F: FnOnce() -> T + Send + 'static,
    T: Send + 'static,
{
    let permit = self.permits.clone().acquire_owned().await?;
    let output = tokio::task::spawn_blocking(operation).await?;
    drop(permit);
    Ok(output)
}

The permit spans exactly the blocking closure. Panic becomes a JoinError; callers must classify it as a defect, not ordinary invalid media.

The closure and output need Send + 'static because they cross into independently scheduled blocking work.

Choose the execution shape

Tokio blocking pool

Good for bounded, moderate synchronous calls integrated with async.

Dedicated CPU pool (for example Rayon)

Good for CPU-parallel algorithms with controlled threads. Avoid nested parallelism when native libraries also create threads.

One-owner thread

Good for thread-confined model/device/native handles. Async tasks send bounded commands.

Subprocess

Good for FFmpeg/untrusted native codecs, hard cancellation, memory limits, crash isolation. Requires pipe draining, signal escalation, temp cleanup, and receipt/status mapping.

Separate service

Good when resource scheduling/fault isolation/scaling differs materially from API workers. Adds network, deployment, queue, auth, and version complexity.

CPU oversubscription

If 16 Tokio blocking tasks each call a library using 16 threads, 256 runnable threads contend on a 16-core host. Symptoms include lower throughput and severe tail latency.

Record/configure:

• process CPU quota/cores;
• runtime worker threads;
• blocking pool/dedicated threads;
• native intra/inter-op threads;
• concurrent jobs;
• NUMA/affinity where material.

Benchmark the entire combination in the deployment container.

Media memory is the primary limit

A compressed 4K video frame expands dramatically. Limit:

• simultaneous decoders;
• frames buffered per stream;
• pixel/sample format;
• dimensions/duration;
• derived thumbnails/windows;
• subprocess resident memory/temp disk.

Admission can use weighted permits representing estimated memory units instead of one permit per job. Validate estimates and enforce hard process/container limits because metadata can lie.

Stream frames/chunks and release promptly. Do not collect a full movie before analysis.

Accelerator scheduling

GPU work needs:

• model residency and loading cost;
• free/reserved device memory;
• input/output tensor bytes;
• batch compatibility/deadline;
• stream/concurrency limits;
• tenant/cost priority;
• kernel completion and buffer lifetime.

One semaphore is a first approximation. A scheduler may group compatible requests, route to resident models, or evict under policy.

Do not run two large models concurrently merely because two host tasks exist. OOM can destabilize the whole worker.

Cancellation

• queued work can cancel before permit acquisition;
• synchronous closures need cooperative atomic checks between chunks;
• native one-shot calls may be uncancellable;
• subprocess can receive terminate then kill;
• device kernels may require waiting/discarding result;
• partially published artifacts must never look complete.

Release permits on every exit via RAII. Record cancel_requested, cancel_observed, and cleanup outcome separately when necessary.

Runnable companion implementation

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example blocking_work
cargo test -p mosaic-runtime

Expected:

blocking total: 5000050000; permits: 2

The example uses BoundedExecutor::run_blocking; library tests prove result return and permit release.

Failure investigation: API freezes during thumbnails

1. confirm thumbnail decoder runs on async worker;
2. inspect CPU/runtime poll latency;
3. move it to bounded blocking/dedicated worker;
4. cap dimensions and concurrent decodes;
5. detect native thread oversubscription;
6. load-test unrelated endpoint latency;
7. consider subprocess isolation for hostile codecs.

Failure investigation: GPU OOM despite request limits

Count model weights, allocator reserve/cache, KV cache, concurrent batches, temporary tensors, and fragmentation—not only upload size. Record device memory around admission/completion and reduce resident/concurrent shapes under an explicit routing policy.

Security and performance implications

Security:

• hostile media belongs behind strict size/time/process limits;
• native crashes may require process isolation;
• subprocess arguments must avoid shell interpolation;
• clean temp artifacts and redact paths;
• verify model/runtime artifacts.

Performance:

• measure queue wait, decode/infer time, memory peak, throughput, tail;
• tune total thread hierarchy;
• batch only within deadline/fairness;
• retain resident models when memory/cost supports it;
• avoid unnecessary host-device copies.

Exercises

1. Add cooperative cancellation to chunked hashing.
2. Build a weighted memory permit abstraction.
3. Execute a subprocess with timeout, output byte cap, terminate/kill.
4. Compare one/two/four decode concurrency at fixed memory.
5. Design a two-model GPU residency scheduler.

Solution guidance

Weighted permits reserve validated estimated bytes before decode and release on drop. A subprocess runner uses argument arrays, drains stdout/stderr concurrently with byte caps, and owns cleanup. Report unknown effect when kill races completion.

Check your understanding

• Does spawn_blocking cap blocking concurrency?
• Can abort stop a running blocking closure?
• Why can one task equal many native threads?
• What often dominates video capacity?
• When is a dedicated owner thread preferable?

Completion checklist

• Runtime workers execute only short/nonblocking polls.
• Blocking/CPU work is bounded separately.
• Native thread counts are included in capacity.
• Media limits include decoded memory.
• Accelerator residency/batching/cancellation are explicit.
• I ran the blocking example and permit-release test.

Authoritative references

• tokio::task::spawn_blocking
• Tokio CPU-bound tasks and blocking code
• Rayon documentation
• std::process::Command