Media Caching, Batching, Memory, and Throughput

An AI media pipeline can be correct for one request and still be an unsafe production system. Decode amplification can exhaust host memory before inference starts. One unusual video can pad an otherwise efficient batch into an accelerator out-of-memory failure. A popular asset can trigger a hundred identical cache fills. A cache hit can reveal that another tenant processed the same private prompt. A queue can increase throughput while silently spending the entire latency budget.

These are not independent performance details. Cache identity determines what work may be reused. Batch compatibility determines what work may execute together. Memory reservation determines whether the batch may enter the device. Deadline and fairness policy determine which compatible requests should execute together. Measurements determine whether any estimate was honest.

This chapter builds one bounded Rust planner for that decision:

request envelope
      │
      ├── validate identity, shape, deadline, and estimates
      │
      ├── derive tenant-scoped cache key
      │       ├── exact hit ───────────────────────► return artifact
      │       └── miss/bypass
      │
      ├── reject impossible deadline/padding/resource requests
      │
      ├── partition by exact compatibility
      │
      ├── apply batch size, tenant fairness, and compute limits
      │
      ├── reserve resident + item + workspace + headroom bytes
      │
      └── emit a deterministic dispatch plan and receipts
      ▼
accelerator worker

The planner does not run a model and does not claim that estimates are measurements. It is the admission seam in front of a runtime. A production worker can replace its fixture estimates with values learned from pinned-release benchmarks while preserving the same contracts.

Learning objectives

By the end you will be able to:

distinguish encoded, decoded, normalized, embedding, prefix, model, intermediate, and output caches;
construct a cache key that binds content, tenant, transform, model, output contract, and revocation generation;
explain why a content digest alone is neither a correctness key nor a security boundary;
choose TTL, invalidation, eviction, negative-cache, and revocation behavior;
stop cache stampedes with single-flight ownership and reject stale completions with generation fencing;
distinguish offline, static, dynamic, sequence, and continuous batching;
define exact compatibility across modality, release, numeric format, device, shape bucket, output contract, and security domain;
quantify padding waste and understand when ragged execution is a real alternative;
reserve host and device memory with checked arithmetic and explicit allocator headroom;
separate allocated, reserved, resident, workspace, pinned-host, and untracked device memory;
schedule against readiness, queue delay, service estimate, deadline, compute budget, and tenant fairness;
use Little’s Law without confusing a stable average with a tail-latency guarantee;
design a benchmark matrix that finds a throughput/latency frontier rather than one flattering number;
investigate OOM, cache contamination, stampedes, head-of-line blocking, starvation, and overload from receipts.

Prerequisites and dependency order

Complete quantization, memory, batching, and device placement for tensor storage and KV-cache arithmetic. Complete the image, audio, and video ingestion chapters before estimating decoded working sets. Complete video generation funnels before admitting expensive generation stages.

This chapter intentionally comes after the modality implementations. An abstract media_bytes estimate is not enough. Image decoders reserve pixels and conversion buffers; audio works in frames, channels, resampler delay, and FFT scratch; video combines compressed packets, decoded frames, temporal windows, and device tensors. The runtime must receive estimates derived from those real contracts.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example media_runtime_plan

The pinned fixture reports:

input requests                   7
tenant-scoped cache hits         1
cache misses                     6
scheduled requests               6
dynamic batches                  4
largest batch                    3
rejections                       0
peak device reservation          475,634,074 bytes
single-flight generation         1
follower coalesced               true
completion became cache hit      true
plan SHA-256                     f991d9c106dc3e19f5c0c947f561bca90a85a3268035bb2acc7a7535b38a8484

The six misses become one three-image batch, one remaining image batch forced by the per-tenant cap, one audio batch, and one video batch. Although the image requests share a model, each cache key remains tenant-scoped. Audio and video cannot join image work merely because every tensor fits on the same device.

Run the focused gates:

cargo test -p mosaic-runtime --all-features media_runtime
cargo test -p mosaic-runtime --all-features --example media_runtime_plan

The implementation is crates/mosaic-runtime/src/media_runtime.rs; the executable is crates/mosaic-runtime/examples/media_runtime_plan.rs.

Start with work reuse, not a generic cache

“Add a cache” hides several different objects and invalidation rules.

Layer	Example value	Usually keyed by	Main invalidation cause
encoded artifact	original PNG, WAV, or MP4 bytes	source digest and authorization scope	deletion, retention, legal hold, changed access
decoded media	oriented RGBA pixels, PCM frames, decoded luma	source + exact decoder and decode policy	decoder/security update, orientation or channel policy
normalized tensor	cropped image, log-mel window, token IDs	decoded identity + preprocessing release + shape/dtype	processor, vocabulary, padding, dtype, device contract
embedding	finite vector	normalized input + model release + pooling/output contract	model, adapter, pooling, normalization, dimension
text/prefix state	tokens, prompt embeddings, KV blocks	exact prefix + template/tokenizer/model/runtime isolation	any token/template/model/adapter/runtime change
intermediate generation	latent, storyboard, checkpoint	request + seed + scheduler + stage releases + step	changed stage, branch, precision, scheduler, safety policy
model output	classification, transcript, image, video	complete request + all releases + output contract	model/provider drift, decoding policy, safety or schema change
resident model	weights, graph, compiled kernels	exact artifact/runtime/device identity	release change, device capability, compiler/runtime change

These layers must not share one untyped namespace. A digest that names encoded PNG bytes does not name oriented pixels. Oriented pixels do not name the crop and normalization used by a vision encoder. An embedding does not remain valid after changing pooling even if model weights stay fixed.

There are also values that should not be cached:

an authorization decision longer than its policy evidence remains valid;
decrypted media in a shared process without an isolation and cleanup contract;
partially validated decoder output;
a non-idempotent tool result;
an error whose cause may recover immediately;
nondeterministic generation presented as though one key implies one canonical answer;
private prompt state in a cross-tenant prefix cache without explicit isolation.

Caching is a claim: “under this complete identity and policy, this value may replace recomputation until this invalidation condition.” Write that claim before choosing a library.

Derive a complete, domain-separated cache key

The companion hashes framed fields rather than concatenating strings:

K = SHA-256(
      domain = "mosaic.media-cache-key.v1",
      tenant_id,
      cache_scope,
      policy_generation,
      cache_layer,
      content_sha256,
      transform_release_sha256,
      model_release_sha256,
      output_contract
    )

Length framing matters. Without it, ("ab", "c") and ("a", "bc") have the same naïve concatenation. Domain separation prevents the same fields from being confused with an artifact or plan identity from another protocol.

Every field answers a different question:

tenant_id is the default privacy boundary;
cache_scope identifies a revocable namespace inside that boundary;
policy_generation invalidates all earlier entries without enumerating them;
cache_layer prevents an encoded artifact from aliasing a normalized tensor;
content_sha256 binds the exact source or complete request material;
transform_release_sha256 binds decode, normalization, crop, tokenization, or feature logic;
model_release_sha256 binds weights and model behavior;
output_contract binds dtype, shape, schema, pooling, coordinate space, and semantic version.

The implementation changes the key when any of those facts changes. It accepts only lowercase 64-hex digests and bounded identifiers. A malformed key input is an error, not a fallback to a weaker key.

Why not key only by URL?

URLs can return different bytes over time, vary by authorization, redirect, or be reused. A versioned object locator plus observed content digest can be evidence; the URL alone is not content identity.

Why not key only by content digest?

The same bytes decoded under two image libraries, two EXIF policies, or two color assumptions can produce different pixels. The same pixels normalized for two encoders produce different tensors. The same tokens under two model releases produce different KV state. Content is necessary but insufficient.

Some products deliberately share public assets. Do not achieve that by omitting the tenant field. Use a separately authorized public cache namespace with its own encryption, retention, revocation, metrics, and timing-leak analysis. Moving an object into that namespace should be an auditable policy operation.

Current vLLM prefix-caching documentation describes including tokens, parent hashes, multimodal input hashes, LoRA-related values, and cache salts. It specifically supports salt-based isolation to reduce timing inference across shared environments. The general rule extends beyond LLM KV blocks: reuse identity must include every behavioral dependency and the security domain allowed to observe the reuse.

Invalidation, expiration, eviction, and revocation are different

These mechanisms are often collapsed into “TTL,” but they solve different problems.

Invalidation says an entry is no longer correct. A model release changed, an authorization was removed, or a preprocessing vulnerability was fixed.

Expiration limits how long an entry may be returned even without a known change. TTL is a freshness and risk bound, not proof that the value stays correct throughout that interval.

Eviction frees capacity. An evicted entry might still be valid; it simply is no longer worth keeping under the current policy.

Revocation is a policy event that must stop future returns promptly. Advancing cache_policy_generation, deleting the old namespace, and rejecting old-generation lookups makes revocation part of key identity. A background deletion alone leaves a race where stale entries may still be returned.

Use both TTL and explicit generation invalidation. TTL bounds unknown drift. Generation invalidates known drift immediately.

Negative caching

A negative cache stores failures such as “object does not exist” or “decoder rejected this artifact.” It can protect a dependency from repeated invalid requests, but it can also prolong an outage or authorization error.

Classify failures:

permanent under exact content and release: malformed bytes, unsupported schema;
policy-dependent: unauthorized, revoked, quota exceeded;
transient: timeout, overload, temporary provider failure;
caller-specific: malformed request, cancelled job.

Only the first class is a natural content-keyed negative cache candidate. Give transient failures a short bounded backoff, not a long success TTL. Never let one tenant’s authorization failure become another tenant’s answer.

Capacity must be weighted

One decoded video window can outweigh thousands of small embeddings. Entry-count capacity is not a memory capacity. Use measured or checked value weights and track key/metadata overhead separately. The Rust Moka async cache, for example, supports weighted capacity, TTL/TTI, invalidation, and coalesced initialization. A production choice still needs workload-specific policy; a capable cache library cannot invent the correct key or security boundary.

Stop cache stampedes with ownership and fencing

Suppose 100 requests miss the same popular image embedding. If every request runs decode and inference, the cache amplified load precisely when it was cold.

A single-flight coordinator gives one request a generation lease:

miss 1 ──► leader(generation 41) ──► compute ──► publish
miss 2 ──► follower(41) ───────────► await ────► read
miss 3 ──► follower(41) ───────────► await ────► read

Coalescing is not enough. Imagine the leader times out, a new leader begins generation 42, and the old worker later finishes. Without fencing, generation 41 can overwrite the newer result.

generation 41 starts ──► lease expires/aborts
generation 42 starts ──► becomes current owner
generation 41 finishes ──► REJECT: stale lease
generation 42 finishes ──► publish, then wake followers

The companion’s CacheCoordinator compares cache key, request owner, and monotonically increasing generation on complete and abort. A stale completion returns StaleGenerationLease.

A production coordinator also needs:

bounded waiter count and follower deadline propagation;
leader cancellation policy;
a lease timeout based on stage cost;
failure fan-out semantics;
retry jitter;
metrics for leaders, followers, wait time, failure, stale completion, and regeneration;
distributed ownership if workers do not share process memory;
a durable or reconstructible record if generation itself is a durable job.

Do not hold a global mutex while computing. Synchronize ownership metadata; run the expensive future outside it. Libraries with atomic async initialization can solve the in-process case. A distributed lease needs a store with atomic compare-and-set and an owner generation, not “check-then-insert.”

Batching is a family of schedulers

Offline batching

The complete dataset is known. Sort and bucket globally, maximize utilization, and write outputs as a durable batch job. Latency per item matters less than total makespan and retry granularity.

Static online batching

Collect exactly N requests before dispatch. It is simple and efficient at high arrival rates but can wait forever under low traffic unless it also has a timer.

Dynamic batching

Dispatch when the batch is full or its bounded delay expires. NVIDIA Triton’s dynamic batcher documentation describes maximum batch size, queue delay, priorities, timeouts, and preferred sizes. It recommends measuring the default behavior and adding delay only while latency remains within budget.

Sequence batching

Stateful streams must remain associated with the correct model instance and correlation identity. Mixing steps from unrelated sequences without state routing corrupts results.

Continuous batching

Autoregressive decoding has unequal sequence lengths. A static batch wastes capacity when short sequences finish. Continuous batching admits new sequences between decoding steps. KV memory and per-step token budget become the scheduler’s core resources. The PagedAttention paper connects paged KV memory management with higher serving concurrency by reducing fragmentation and duplication.

Image encoders and fixed-window classifiers commonly use dynamic request batching. Text generation commonly needs continuous token scheduling. Video generation may use job-level stage scheduling plus fixed batches inside a denoising stage. “Batch size” does not identify one universal mechanism.

Exact compatibility comes before optimization

The companion partitions misses by:

modality
model release digest
transform release digest
output contract
numeric format
device class
security domain
width bucket
height bucket
temporal bucket

Items with different keys do not join. This is stricter than checking that tensors happen to have the same byte length.

Two tensors can share shape but differ in:

RGB versus BGR channel meaning;
sample rate or mel-bin definition;
frame versus token temporal units;
normalization mean and standard deviation;
model adapter or LoRA;
FP16 storage versus BF16 compute;
CPU versus CUDA kernels;
classification versus embedding output;
confidential versus shared execution zone.

Add every fact the worker requires for one model invocation. If a backend genuinely supports mixed adapters, ragged shapes, or heterogeneous control inputs, version that capability in the worker contract and test demultiplexing. Do not weaken the key based on hope.

Cross-tenant batching

The fixture permits tenants to share an inference batch inside one security_domain, while cache keys remain tenant-specific. This can be safe only if:

authorization occurred before admission;
raw inputs and decoded buffers are slot-scoped;
the model does not carry request state across slots;
output demultiplexing uses stable request ownership;
logs and errors do not expose another slot;
cancellation cannot return or reuse the wrong buffer;
timing and aggregate metrics satisfy the product’s threat model.

Products with stronger isolation requirements should assign a per-tenant security domain, forcing separate batches. This reduces utilization but expresses the real constraint.

Bucket shapes and quantify padding waste

Most accelerators prefer regular tensors. A 224×200 image placed in a 224×224 bucket has:

actual elements = 224 × 200 = 44,800
padded elements = 224 × 224 = 50,176
waste           = (50,176 - 44,800) / 50,176
                ≈ 10.714%

The planner records actual and padded elements and rejects a request when:

padding_waste_bps > max_padding_waste_bps

It uses integer basis points so the admission decision does not depend on floating-point rounding.

Padding waste affects more than input bytes. A padded video can expand spatial activations for every frame and every layer. A padded audio window may increase attention quadratically in temporal length. Estimate the model’s real cost curve; input-element ratio is an admission proxy, not a universal compute predictor.

Alternatives include:

more buckets, trading less padding for more queue fragmentation;
crop/resize policies, changing semantic content and therefore transform identity;
ragged tensors or packed sequences, if the backend and kernels truly support them;
per-item execution for rare shapes;
a specialist route for long audio or large video;
offline processing where enough similar shapes accumulate.

The optimal bucket set depends on the observed shape distribution and measured kernel behavior.

Build a deadline-aware dynamic batch

For each validated miss or cache bypass, the companion:

calculates actual and bucket elements with checked multiplication;
rejects excess padding;
calculates the earliest possible finish;
rejects impossible deadlines before queuing;
rejects an individual item larger than compute or memory capacity;
partitions by compatibility;
sorts compatible work by deadline, ready time, and request ID;
greedily admits while batch, tenant, compute, host, and device bounds hold;
defers an item that cannot join the current batch into a later batch;
calculates the last safe dispatch time from every item’s deadline;
dispatches no later than that time and no later than the configured queue-delay target;
hashes the complete stable plan.

For item i:

earliest_ready_i  = max(now, ready_at_i)
earliest_finish_i = earliest_ready_i + service_i
latest_start_i    = deadline_i - service_i

For batch B:

all_ready       = max(earliest_ready_i)
latest_start    = min(latest_start_i)
dispatch_target = all_ready + max_queue_delay
dispatch_at     = min(dispatch_target, latest_start)
service(B)      = max(service_i)    // fixture estimate, not a universal law

The maximum service estimate is appropriate only for this fixture’s same-kernel shape bucket. Real batch service time must come from measurements indexed by release, device, shape, and batch size. It is often sublinear in item count until a memory or kernel threshold, but it is not free.

Fairness is a capacity constraint

The fixture limits one tenant to two items per batch. Three urgent requests from tenant A and one from tenant B therefore form a three-item batch with two A slots and one B slot; A’s third request moves to a second batch.

This is a local mechanism, not a complete fair queue. A production scheduler may need:

per-tenant queued-item and byte limits;
deficit round-robin or weighted fair queueing;
reserved capacity for interactive traffic;
separate bulk and online pools;
maximum consecutive dispatches per tenant;
cost-weighted quotas rather than request counts;
aging that raises long-waiting work without violating hard deadlines.

Priority without admission control can starve ordinary work. Fairness without cost weighting lets one “request” contain an enormous video. Define the resource being shared.

Reserve memory before dispatch

The companion computes host and device reservations independently:

base =
    resident_bytes
  + sum(item_bytes)
  + workspace_bytes_per_batch

headroom = ceil(base × headroom_bps / 10,000)
reserved = base + headroom

admit only if reserved ≤ capacity

All addition and multiplication is checked. Overflow is an error, never a wrapped small reservation.

Resident bytes

Resident device memory includes model parameters, constant buffers, compiled graphs, runtime state, and any preallocated cache pool. Resident host memory may include mapped weights, decoder state, indexes, and process baseline. Measure both after warmup.

Item bytes

Count the peak live request-specific values, not only input tensors:

encoded input while decoding;
decoded image/audio/video;
normalized tensor;
intermediate activations;
KV cache or diffusion latents;
output buffers;
encoder staging;
concurrent copies during format conversion.

Lifetimes matter. If decoded pixels remain alive while a device tensor is built, their peaks overlap. A streaming decoder can reduce overlap only if the next stage consumes chunks and does not secretly materialize the full input.

Workspace

Libraries allocate algorithm scratch, attention workspace, FFT buffers, convolution plans, graph capture pools, and encoder buffers. Workspace can change abruptly with batch or shape. Benchmark the selected kernel path.

Headroom

Headroom covers estimation error, allocator fragmentation, asynchronous lifetimes, driver/runtime allocations, and unmodeled libraries. It is not permission to guess. If actual peaks repeatedly consume headroom, update the estimate.

PyTorch’s CUDA memory documentation separates tensor memory from memory reserved by its caching allocator. Its memory snapshot guide also warns that allocations made directly through CUDA or other libraries may be invisible to the PyTorch profiler. The lesson is runtime-independent: one allocator’s telemetry is not the device’s complete memory truth.

Host, pinned host, and device are separate pools

Pinned host buffers improve asynchronous transfer but are non-pageable and should be bounded. Decoded video may exhaust ordinary host memory while device reservation still looks safe. A remote inference service may have low application memory but high network buffering. Track each pool with its own capacity, not one combined byte number.

Why “free VRAM right now” is not an admission plan

Two workers can observe the same free capacity and both allocate it. Concurrent kernels can retain buffers past a host-side function return. Memory may be fragmented into blocks that cannot satisfy one large allocation. Admission requires a process-owned reservation ledger coordinated with actual allocation and release.

The plan’s receipt is the requested reservation. The worker must reconcile it with measured peak and release it exactly once on success, failure, timeout, or cancellation.

Throughput, latency, concurrency, and utilization

Throughput is completed useful work per unit time. Latency is a distribution of time per request. Concurrency is work simultaneously in the system. Utilization is the fraction of a resource’s capacity occupied. Increasing one can harm another.

Little’s Law for a stable system is:

L = λW

where:

L is average items in the system;
λ is average completed arrival rate;
W is average time in the system.

If a service completes 40 requests/s and mean end-to-end latency is 250 ms:

L = 40 × 0.250 = 10 requests

This is an average conservation relationship, not a promise that concurrency 10 satisfies p99. It assumes a stable observation interval. If arrivals exceed sustainable completions, the queue grows and the measured system is not in steady state.

Separate the latency budget

Record at least:

end-to-end
  = ingress
  + authorization
  + cache lookup
  + queue wait
  + decode/preprocess
  + host-to-device transfer
  + model service
  + postprocess/encode
  + persistence
  + egress

For streaming generation, also record time to first output and inter-output latency. An average model-service time cannot explain a queueing regression.

Utilization near one is dangerous

As utilization approaches capacity, small arrival or service-time variance produces large queues. A GPU at 99% can look economically excellent while interactive p99 becomes unacceptable and recovery traffic causes collapse. Keep explicit overload policy:

bounded queues;
admission before expensive decode;
deadline-aware rejection;
client retry budgets with jitter;
load shedding by traffic class;
concurrency limits at each scarce dependency;
degraded routes that preserve semantics honestly;
autoscaling signals based on queued cost and latency, not CPU alone.

Google’s SRE chapter on handling overload is a useful reminder that retries and poorly coordinated clients can amplify load. Backpressure must reach the source; moving unlimited work into an internal queue only hides overload.

Measure a frontier, not one batch size

For each exact model/runtime/device release, benchmark a matrix:

batch size:       1, 2, 4, 8, ...
shape bucket:     observed production buckets
numeric format:   supported storage/compute combinations
queue delay:      0, small bounded values
concurrency:      1..saturation
cache state:      cold, warm, mixed
input class:      text/image/audio/video distributions
output length:    especially for autoregressive/generative work

At every point record:

completed items/s and useful tokens/frames/s;
p50, p90, p95, p99, and maximum queue/service/end-to-end latency;
peak allocated and reserved host/device bytes;
decode and transfer overlap;
padding ratio;
cache hit, byte-hit, and compute-saved ratios;
failure, timeout, cancellation, and retry counts;
quality or numerical drift if kernels/dtypes change;
power or cost per successful unit where available.

Warm up model loading, kernel selection, graph compilation, allocator pools, and filesystem caches, but also measure cold starts because rollouts and scale-out create them. Randomize trial order or repeat cycles so thermal and background effects do not bias one configuration. Pin release, hardware, drivers, clocks/power policy where controllable, and dataset identity.

Plot throughput against p99 latency and peak memory. Choose a Pareto point that satisfies product budgets. The largest successful batch in an isolated benchmark is rarely the correct online default.

Observability contract

Every planned or rejected request needs bounded, non-secret dimensions:

request and trace identity;
tenant pseudonym or bounded internal ID, never raw prompt;
cache layer, disposition, and key prefix safe for correlation;
release and compatibility identity;
queue insertion, ready, dispatch, start, first output, finish;
batch ID, size, actual/padded elements, and tenant counts;
requested and measured host/device peaks;
compute estimate and actual unit count;
deadline slack at admission and dispatch;
typed rejection or failure;
single-flight role, generation, wait duration, stale-completion count.

Avoid high-cardinality prompt, URL, or digest labels in metrics. Put bounded identities in traces or structured logs with retention controls. Metrics aggregate by modality, release, bucket, route, cache layer, and outcome.

Reconcile three layers:

planner estimate ──► worker allocation ledger ──► device/OS measurements

A planner that is consistently wrong should fail an operational objective even when requests still complete. Estimate error is an early warning.

Failure investigations

Investigation 1: cache hits return the wrong embedding after a rollout

Symptom: no errors, but retrieval quality drops only on warm entries.

Likely cause: key bound content and model name but not exact revision, pooling, or preprocessing.

Evidence: compare cache keys and receipts across cold/warm executions. Recompute one warm item under both releases.

Fix: bind transform, model digest, output contract, and policy generation. Advance generation on rollout. Add a test that each behavioral change changes the key.

Investigation 2: one tenant can infer another tenant’s prompt

Symptom: repeated private prefix has lower latency for a different tenant.

Likely cause: cross-tenant prefix or output caching without salt/scope isolation.

Fix: tenant-scoped keys or explicitly authorized trust-group salts; consider disabling shared reuse for sensitive routes. Treat latency as an observable side channel.

Investigation 3: cold start causes an inference storm

Symptom: replicas scale out, caches are empty, dependency traffic multiplies, and latency rises.

Likely cause: check-then-compute cache fill with no single-flight and synchronized retries.

Fix: generation ownership, bounded followers, jitter, warmup of high-value public entries, and fleet-aware rollout. Verify only one leader computes a key.

Investigation 4: an old worker overwrites fresh cache data

Symptom: value reverts after a timeout/retry race.

Likely cause: completion is authorized only by key, not generation lease.

Fix: compare-and-set current generation. Reject stale completion. The companion has a regression test for exactly this sequence.

Investigation 5: p99 rises after “throughput optimization”

Symptom: items/s improves, queue wait dominates p99.

Likely cause: queue delay or batch-size target consumes interactive latency slack.

Fix: deadline-aware dispatch, separate online/bulk routes, benchmark the complete latency distribution, and reduce delay until the service-level objective holds.

Investigation 6: GPU OOM occurs below the planner’s estimate

Symptom: tensor estimate fits; allocator fails.

Likely cause: omitted resident/workspace/encoder memory, overlapping lifetimes, fragmentation, or allocations outside the measured allocator.

Fix: capture allocator/device evidence, identify peak overlap, add missing pool, and calibrate headroom. Do not “fix” by catching OOM after admission and retrying the same batch.

Investigation 7: small requests wait behind a long video

Symptom: low-cost work misses deadlines despite spare average throughput.

Likely cause: FIFO uses request count instead of cost and route.

Fix: cost-aware fair queueing, separate resource pools or traffic classes, preemptible stage boundaries, and bounded video concurrency.

Investigation 8: adding shape buckets lowers throughput

Symptom: padding improves but batches are smaller and queue waits rise.

Likely cause: over-partitioning fragments arrivals.

Fix: measure bucket occupancy and merge nearby shapes whose extra padding costs less than lost batch efficiency. Bucket design is a distribution problem.

Investigation 9: cancellation leaks capacity

Symptom: admission gradually rejects everything although no work is active.

Likely cause: reservation released only on success or follower cancellation cancels the shared leader incorrectly.

Fix: ownership-bound RAII reservations, terminal paths for success/failure/cancel/timeout, and separate follower lifetime from leader computation policy.

Investigation 10: hit rate is high but cost does not fall

Symptom: 90% entry hit rate, nearly unchanged accelerator utilization.

Likely cause: many hits are tiny while large video/tensor misses dominate bytes and compute.

Fix: report entry-hit, byte-hit, estimated-compute-saved, and latency-saved ratios by layer. Optimize the expensive miss distribution, not one vanity percentage.

Production implementation choices

The companion uses deterministic in-memory structures so every rule is inspectable. Production systems can replace pieces independently:

Moka or a custom bounded map for process-local values;
object storage/CDN for immutable encoded artifacts;
Redis or another atomic store for distributed metadata and short leases;
durable jobs for long generation;
Triton or a model runtime’s scheduler for supported fixed-shape inference;
vLLM/mistral.rs-style continuous scheduling for autoregressive models;
a Rust admission service in front of heterogeneous pools.

Keep the contracts:

cache identity
single-flight generation ownership
batch compatibility
resource reservation
deadline/fairness policy
typed rejection
measurement reconciliation

Do not duplicate scheduling blindly. If the inference server already dynamically batches, a second upstream batcher can add delay without improving device work. The upstream layer may still perform authorization, cache lookup, cost admission, and route selection, then send individual requests to the server’s scheduler.

Project milestone: the mixed-media admission gateway

Extend the companion into a service milestone:

accept strict image/audio/video work envelopes;
derive decode and tensor estimates from the earlier modality policies;
authorize tenant and cache scope;
check a weighted cache;
coalesce misses with generation leases;
enqueue a bounded compatibility record, never raw unvalidated bytes;
dynamically dispatch under deadline, fairness, compute, host, and device limits;
hand a reservation token to the worker;
reconcile measured usage;
publish output only under the current generation;
release every resource on every terminal path;
expose a replayable plan receipt and redacted telemetry.

Acceptance tests should include:

exact cache hit and six kinds of key-changing drift;
tenant-isolation timing policy;
100 concurrent same-key misses with one computation;
stale worker completion;
queue full and deadline expiry;
incompatible modality/release/shape/security records;
padding rejection;
host and device OOM before allocation;
checked integer overflow;
cancellation during queue, compute, and publish;
worker crash and lease recovery;
deterministic replay of one scheduler snapshot;
load test proving bounded queue and stable tail latency below target rate;
overload test proving explicit shedding above sustainable rate.

The exit artifact is not “a cache and batcher.” It is a resource-accounted admission gateway whose decisions can be explained from stable input evidence.

Exercises

1. Key drift matrix

Add table-driven tests that mutate cache scope, content, transform, model, layer, output contract, and policy generation one at a time. Assert every mutation changes the key.

2. Public cache namespace

Design a separate key type for explicitly public artifacts. Identify the authorization event that allows promotion and how revocation works.

3. Weighted cache capacity

Implement an in-memory cache bounded by total encoded/decoded bytes rather than entry count. Account for metadata overhead and reject a single value larger than total capacity.

4. Follower deadline

Add a follower whose deadline is earlier than the leader’s estimated completion. It should stop waiting without cancelling a leader still useful to other followers.

5. Lease expiry race

Use manual time to expire generation 1, begin generation 2, then finish generation 1. Prove the old result cannot publish.

6. Negative-cache classification

Create an enum for permanent-content, policy, transient, and caller failures. Permit caching only under an explicit per-class policy.

7. Bucket optimizer

Given a fixture distribution of image sizes, evaluate several bucket sets. Report total padded elements and the number of independent queues.

8. Ragged capability

Add a worker release that explicitly supports ragged audio. Keep it incompatible with the padded worker and test output demultiplexing.

9. Cost-weighted fairness

Replace item-count fairness with deficit round-robin over compute units. Prove a large video cannot starve small images and small images cannot starve video forever.

10. Deadline aging

Generate requests with varied readiness and deadlines. Demonstrate that a later-arriving urgent request can dispatch first without reordering responses incorrectly.

11. Reservation RAII

Create a host/device reservation guard whose Drop returns capacity. Test success, typed failure, panic isolation, timeout, and cancellation.

12. Measured reconciliation

Record estimate and measured peak for each bucket. Fail a calibration check when p99 actual usage exceeds the estimate plus declared headroom.

13. Little’s Law

For 75 completed requests/s and 320 ms mean time in system, calculate average concurrency. Then explain why the result cannot determine p99 capacity.

14. Benchmark frontier

Benchmark at least four batch sizes and three concurrency levels. Plot throughput against p99 and peak device memory. Choose a point under explicit budgets.

15. Overload drill

Drive arrival rate above sustainable completion rate. Prove the queue stays bounded, rejections are typed, and accepted-request tail latency recovers after load falls.

16. Full admission receipt

Persist one plan, one batch, one worker reservation, one measured-usage record, and one cache publication lease. Verify every identity joins and no raw media appears in telemetry.

Solutions and review notes

A passing test changes one field while holding all others fixed. A combined mutation cannot identify an omitted dependency.
Public promotion is a policy transition, not a caller-selected scope string. Use its own generation and retention controls.
Use checked addition for key/value/metadata weights. Evict until the new value fits; reject it if it can never fit.
The follower owns its wait, not the shared work. Cancellation propagation must distinguish these lifetimes.
Completion must compare key, owner, and generation atomically. Wall-clock recency alone is racy.
Malformed immutable content can have a content/release-bound negative entry. Timeouts and overload should receive short retry policy, not long cached answers.
Fewer buckets improve arrival aggregation; more buckets reduce padding. Optimize measured end-to-end cost under latency constraints.
Ragged support belongs in compatibility identity. Test lengths, offsets, and per-request output ownership.
Charge real estimated cost and add bounded aging. Fairness requires a defined unit.
Earliest-deadline-first works only among work that is ready and feasible. Preserve request IDs through output demultiplexing.
A guard should be created only after atomic reservation succeeds and must release exactly once.
Recalibrate by pinned release/device/bucket. Do not combine incompatible measurements.
L = 75 × 0.320 = 24 average requests. Averages omit variance and tails.
Report repetitions and confidence/variance, not only the best run. Reject points that violate either latency or memory.
Sustainable systems shed before unbounded queuing. Retry traffic belongs in the drill.
Join on stable IDs: request → cache key/flight → batch → reservation → execution → artifact. Redaction tests are part of acceptance.

Primary references

Definition of done

You are finished when you can demonstrate all of the following:

cache keys change across every correctness, release, contract, tenant, and revocation boundary;
cache fills coalesce and stale workers cannot publish;
incompatible work never shares a batch;
padding, deadline, compute, host, and device failures occur before expensive execution;
fairness and queue delay are explicit policy;
all resource arithmetic is checked;
plan order and identity are deterministic for one scheduler snapshot;
worker allocation is reconciled against the reservation;
queues and waiters are bounded under overload;
benchmark evidence identifies a throughput/latency/memory frontier;
telemetry can explain one hit, miss, rejection, batch, OOM prevention, cancellation, and stale completion without exposing source content.

That is the difference between “the GPU usually stays busy” and an AI media runtime whose reuse, co-scheduling, and capacity decisions can be trusted.