Tower Middleware, Limits, Timeouts, and Request Identity

Middleware is executable policy around a service. In an AI backend it decides whether a request is small enough, authenticated, within quota, traceable, and still inside its deadline before the application allocates scarce work.

Tower models each policy as Service<Request> -> Future<Response>. Axum routers are Tower services, so the same abstraction covers route handlers, retries, timeouts, concurrency, tracing, and tests.

Learning objectives

• understand Tower’s Service and Layer roles;
• reason about layer order in both request and response directions;
• set request-body and deadline limits before expensive work;
• generate and propagate request identity;
• distinguish transport timeout from run deadline and cancellation;
• keep middleware error responses in the public envelope;
• select rate, concurrency, and load-shedding policy;
• test middleware without a live network;
• debug missing IDs, bypassed limits, and timeout resource leaks.

Dependencies and mental model

Depends on the Axum boundary, Futures, cancellation, and backpressure.

request
  │
  ▼
outer layer A
  │ request path
  ▼
inner layer B
  │
  ▼
handler
  │
  ▼
inner layer B
  │ response path
  ▼
outer layer A

Ordering is semantic. If identity is generated inside tracing, the span cannot see it. If body limiting is inside an extractor that buffers the body, the allocation already happened. If response normalization is inside the layer that creates a timeout response, it cannot normalize that timeout.

Service versus Layer

A Tower Service receives a request and eventually produces a response or service error. It has readiness and call phases:

poll_ready → call(request) → response future

Readiness lets middleware implement capacity/load behavior without accepting unlimited calls. Callers must obey the service contract: do not call a service that is not ready.

A Layer is a constructor/decorator:

Layer<InnerService> → WrappedService

Layers are reusable configuration. Services contain per-instance state. Axum’s Router::layer wraps routes already added. Repeated router-layer calls can be easy to misread; tower::ServiceBuilder applies layers in the order they are written and makes this stack reviewable. Whichever syntax you choose, write down and test the effective inbound and outbound order.

The companion policy stack

The mosaic-api router applies:

.layer(
    ServiceBuilder::new()
        .layer(SetRequestIdLayer::new(
            request_id_header.clone(),
            request_ids,
        ))
        .layer(TraceLayer::new_for_http())
        .layer(PropagateRequestIdLayer::new(request_id_header))
        .layer(TimeoutLayer::with_status_code(
            StatusCode::REQUEST_TIMEOUT,
            REQUEST_TIMEOUT,
        ))
        .layer(RequestBodyLimitLayer::new(MAX_REQUEST_BYTES)),
)
.layer(middleware::map_response(normalize_middleware_errors))

ServiceBuilder applies its layers in the listed request order. The final router layer wraps the stack with response normalization. The effective request path is:

normalize response wrapper
→ set request ID
→ trace span
→ propagate request ID around every inner response
→ request timeout
→ request body limit
→ route/handler

The response unwinds in reverse. The request ID exists before tracing and limiting. Propagation sees the ID and attaches it to responses. The outer normalizer sees transport-generated 408/413 responses and converts them to the same JSON error shape as handler failures.

This order is a reasoned policy, not a universal recipe. Authentication may need to run before tracing fields are enriched; an edge proxy may set trusted identity; streaming endpoints need body timeouts rather than a response deadline.

Request identity

The companion generator implements MakeRequestId using an atomic process-local sequence:

impl MakeRequestId for SequenceRequestId {
    fn make_request_id<B>(&mut self, _request: &Request<B>) -> Option<RequestId> {
        let value = self.next.fetch_add(1, Ordering::Relaxed);
        let header = HeaderValue::from_str(
            &format!("req-{value:016x}")
        ).ok()?;
        Some(RequestId::new(header))
    }
}

This is deterministic and collision-free within one process lifetime. It is a teaching implementation, not a globally unique identity. Production IDs should remain unique across replicas/restarts and avoid encoding tenant, time precision, host, or secrets unless intentionally designed.

Identity classes should remain distinct:

• request ID: one HTTP attempt;
• trace ID/span ID: distributed telemetry context;
• run ID: durable AI operation;
• task ID: requested logical task;
• idempotency key: client-chosen retry identity;
• artifact/event ID: durable object/event identity.

Do not use a client-supplied x-request-id as proof of identity. SetRequestIdLayer sets an ID when the header is missing; a trusted ingress should strip/replace untrusted correlation headers or the application should use a private internal header. W3C trace context also requires trust and sampling policy.

Body limits

RequestBodyLimitLayer limits request bytes. With a declared Content-Length beyond the limit, it can return 413 before the handler. For streamed/chunked bodies it wraps the body and stops after the limit is crossed.

Item count is insufficient:

1 JSON request × 200 MB string = still one item
1 multipart file × decompression ratio 1000 = bounded wire, unbounded expansion

Define limits at multiple representations:

• wire/request bytes;
• decompressed bytes;
• JSON nesting/string/array counts;
• multipart part count and per-part bytes;
• decoded pixels/audio samples/video duration/frames;
• tokenizer input/output units;
• object-store artifact size;
• queue item bytes.

Apply the cheapest trustworthy limit first. Upload large media directly to object storage with short-lived scoped authorization; enqueue metadata plus verified hash rather than proxying all bytes through the API process.

The companion maps a limit rejection to:

{
  "error": {
    "code": "payload_too_large",
    "message": "request body exceeds the 32768-byte limit"
  }
}

Do not reflect the rejected body.

Timeouts, deadlines, and cancellation are different

A timeout says “allow this stage at most duration D from now.” A deadline says “all remaining work must finish before instant T.” Nested independent timeouts can exceed or fight the caller’s total budget:

request deadline: 2 s
database timeout: 1 s
provider timeout: 10 s  ← impossible budget

Propagate a deadline/remaining budget:

remaining = deadline - now
stage_budget = min(configured_stage_limit, remaining - cleanup_reserve)

The companion’s two-second HTTP timeout protects a thin admission handler. The model run is detached behind 202 and has its own durable run deadline.

A timeout response does not guarantee the underlying world stopped:

• dropping a Rust future cancels only cancellation-safe work;
• a spawned task continues unless explicitly cancelled/joined;
• a provider may have received/charged the request;
• a database statement may continue server-side;
• a subprocess/GPU kernel may continue;
• a durable job must not be cancelled merely because one client disconnected.

Pair timeouts with structured cancellation and idempotency. Reserve cleanup time. Record which boundary timed out.

For streaming/SSE, a whole-response timeout can terminate a healthy long stream. Use separate policies:

• time to headers/first event;
• idle/no-progress timeout;
• maximum run deadline;
• per-write timeout and slow-consumer buffer;
• heartbeat interval;
• disconnect policy (cancel or detach).

Rate, concurrency, and load shedding

These controls solve different problems:

• rate limit: arrivals over time;
• concurrency limit: simultaneous in-flight work;
• queue bound: waiting work;
• body limit: bytes per request;
• timeout/deadline: time;
• load shed: reject when inner service is not ready;
• quota: principal-owned allowance (calls/tokens/cost/storage).

One global concurrency limit can let cheap health/status requests queue behind large admissions. Apply limits by route/resource:

metadata reads       high concurrency, small body
run admission        tenant quota + bounded DB/queue
media finalize       byte/pixel/duration limits
admin operations     separate auth and tiny concurrency
provider calls       provider/model semaphore inside worker
GPU decode/inference device scheduler inside worker

HTTP 429 normally communicates principal rate/quota; 503 communicates temporary service capacity. Retry-After should reflect a real safe policy, and retries require idempotency.

Authentication and authorization placement

Recommended flow:

1. edge connection/header size and coarse rate limit;
2. request identity/tracing shell;
3. strict transport body bound;
4. authenticate from parts/headers;
5. derive principal/tenant/capabilities;
6. authorize route/action;
7. apply per-principal quota;
8. parse/validate body;
9. durable admission.

Exact order depends on whether authentication requires a signed body. Never parse unlimited data to discover the caller. Never place authorization only in a handler that can be bypassed by another route. Redact tokens and avoid high-cardinality principal labels.

Error normalization

Middleware can generate responses before a handler:

• request limit: 413;
• timeout: configured 408;
• authentication: 401;
• authorization: 403;
• rate/quota: 429;
• load shed: 503.

If clients depend on one envelope, an outer response mapper must normalize those statuses while preserving safe headers such as request ID, Retry-After, and authentication challenge. The companion’s normalize_middleware_errors handles 408 and 413.

Be precise: a generic mapper that rewrites every 5xx can erase useful headers, streaming bodies, or handler-specific error codes. Normalize only owned statuses/types and test header preservation.

Observability without leakage

A trace span can safely record low-cardinality fields:

• method;
• route template, not raw attacker-controlled path;
• status class;
• request byte bucket;
• authenticated service tier;
• queue outcome;
• duration;
• timeout/limit class.

Attach request/run IDs to structured logs/traces, not metric labels. Never record authorization headers, cookies, arbitrary query strings, full prompts, multipart bodies, provider payloads, or signed URLs by default.

Tracing layer placement determines which rejections are visible. In the companion, request ID exists before TraceLayer, and body-limit/handler outcomes occur inside it.

Runnable verification

cd rust/rust-ai-engineering
cargo test -p mosaic-api
cargo clippy -p mosaic-api --all-targets --all-features -- -D warnings

The oversized-body test sets Content-Length to MAX_REQUEST_BYTES + 1 and proves:

• handler admission is not reached;
• response status is 413;
• response uses payload_too_large;
• a request ID is still propagated.

The full/closed queue tests verify transport policy is connected to real backpressure rather than a canned success response.

Failure investigation: 413 response has no request ID

1. write the effective inbound layer order;
2. locate which layer generated 413;
3. confirm identity is set outside/before that layer;
4. confirm propagation sees the request header;
5. confirm response normalization preserves the header;
6. add a router-level regression test.

The fix is ordering, not adding an unrelated ID after the fact.

Failure investigation: client got 408 but provider spend continued

1. determine whether the model call ran in the handler, a spawned task, or durable worker;
2. inspect parent/child cancellation ownership;
3. check whether provider request supports cancellation and idempotency;
4. correlate request ID, run ID, and provider operation ID;
5. enforce run deadline independently of HTTP;
6. cancel and join non-durable children with cleanup reserve;
7. if work is durable, document that HTTP timeout detaches rather than cancels.

Do not assume dropping the response future reversed external side effects.

Failure investigation: limit is configured but memory still spikes

1. identify wire, decompressed, buffered, decoded, and queued sizes;
2. verify layer surrounds the body-consuming extractor;
3. inspect reverse proxy and decompression order;
4. find collect, to_bytes, Vec, image decode, and multipart temp behavior;
5. enforce media dimension/duration/ratio limits before full allocation;
6. load-test chunked input and compression bombs.

Alternatives and trade-offs

Router .layer calls

Readable beside routes and supports route-specific stacks. Easy to misunderstand when many calls are reordered.

ServiceBuilder

Central stack and explicit composition. Generic response/error types can become harder to read; route-specific policy may need separate routers.

Edge-only policy

Reduces application work and centralizes controls. Does not protect internal/direct paths or encode domain quota; application still needs defense in depth.

Duration timeouts

Simple local configuration. Lose end-to-end budget across hops/retries. Prefer propagated deadlines for distributed work.

Trust incoming IDs

Preserves client correlation. Allows spoofing/collisions unless ingress trust is established. Store external correlation separately from server identity.

Security and performance implications

Security:

• layer order is a security property and belongs in tests/review;
• bound headers/body/decompression/media before allocation;
• authenticate/authorize every route group, including streams/artifacts;
• distrust incoming IDs and tracing baggage;
• avoid error/body logging;
• per-principal controls prevent one tenant consuming global capacity;
• timeouts need cancellation and external-side-effect analysis.

Performance:

• tracing/logging allocation on every chunk can dominate streaming;
• timeout timers and layer stacks add small per-request overhead—measure, but keep safety;
• global concurrency limits may cause head-of-line blocking;
• rate limit state needs bounded cardinality and distributed semantics;
• proxy/app timeout disagreement creates aborted useful work;
• histograms should distinguish queue, handler, storage, provider, and stream phases.

Exercises

1. Draw the exact request and response order of the companion ServiceBuilder stack.
2. Add a trusted-ingress mode that always replaces external request IDs and stores external correlation separately.
3. Add a one-millisecond timeout around a deliberately slow test service and verify the normalized envelope and request-ID header.
4. Add a per-tenant token bucket plus global concurrency gate; document 429 versus 503.
5. Design policies for a two-hour resumable SSE run without a two-hour unconstrained HTTP future.
6. Load-test chunked bodies whose declared size is absent.

Solution guidance

Test stacks as services with oneshot and deterministic clocks where possible. Identity generation must be outside tracing and every early rejection. For trusted ingress, delete/replace untrusted headers at the boundary and optionally retain a validated external correlation field. A resumable stream reads durable sequenced events, uses idle/write deadlines and bounded buffers, and allows the run to continue independently.

Check your understanding

• How does request order differ from response order?
• Why must identity be created outside the body-limit layer?
• Does a timeout stop a spawned task or provider charge?
• What does 413 limit, and what does it not limit?
• When should overload be 429 versus 503?
• Why is a request ID unsuitable as a metric label?
• Why can a whole-response timeout break healthy SSE?

Completion checklist

• The effective layer order is documented and tested.
• Identity exists on success and early rejection.
• Incoming correlation context has an explicit trust policy.
• Bytes, concurrency, rate/quota, queue, and time have separate bounds.
• Middleware errors use stable envelopes and preserve required headers.
• Deadlines propagate and leave cleanup reserve.
• Timeout cancellation/side effects are understood.
• Streaming endpoints have idle/slow-consumer policy.
• Logs/traces/metrics are bounded and redacted.
• I ran the router tests and Clippy.

Authoritative references

• Tower Service
• Tower Layer
• Axum middleware documentation
• Tower HTTP request-body limit
• Tower HTTP request identity
• Tower HTTP timeout
• Tower HTTP tracing
• W3C Trace Context
• HTTP Semantics