Retry Ownership, Provider Fallback, and Idempotency

Retries are not free reliability. An unowned retry can:

multiply traffic during an outage;
spend the same model budget several times;
repeat an external effect;
cross a privacy boundary through fallback;
return a semantically weaker output under the same contract;
continue after the caller’s deadline;
hide the real number of attempts from traces and evaluation.

A production harness treats retry and fallback as one explicit state machine:

one logical request + one idempotency identity
                 │
                 ▼
        harness-owned dispatch
                 │
        typed observed outcome
                 │
     ┌───────────┼────────────┬───────────────────┐
     ▼           ▼            ▼                   ▼
 complete   retry same    compatible fallback   stop/reconcile

There must be one visible retry owner. Every physical attempt must remain attributable to the same logical request. Fallback is allowed only through prevalidated routes that preserve capability, behavior, data-boundary, output-validation, and idempotency requirements.

This chapter implements the deterministic control plane in mosaic_harness::retry_fallback. The runnable companion is retry_fallback_replay.

Learning objectives

By the end you will be able to:

name one retry owner across SDK, adapter, harness, queue, and workflow layers;
distinguish logical requests, physical attempts, repairs, and user resubmissions;
classify retryable, non-retryable, and indeterminate outcomes;
bind every attempt to one request and idempotency identity;
validate fallback routes before runtime rather than after an outage;
preserve capability, behavior class, policy, and data boundary across fallback;
calculate bounded exponential backoff and honor bounded server delay hints;
combine attempt, cost, route, and deadline budgets;
reason about streamed partial output and unknown provider acceptance;
prevent retry storms, synchronized recovery, and circuit-breaker misuse;
durably resume without repeating an unrecorded or uncertain attempt;
replay a final retry/fallback receipt and detect schedule or route drift.

Prerequisites

Complete the run-envelope, provider-neutral capability, typed-tool, agent-loop, and production durability chapters first.

The run envelope provides the deadline and budgets.
Capability contracts say what a route actually supports.
Typed tools classify external effects and reconciliation.
The agent loop decides whether another model action is useful.
Durable jobs provide leases, idempotency records, and restart ownership.

Retry is transport/orchestration recovery. Repair is a new semantic generation in response to a validation defect. Do not merge their budgets.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness retry_fallback
cargo test -p mosaic-harness --example retry_fallback_replay
cargo run -q -p mosaic-harness --example retry_fallback_replay

Pinned result:

baseline                            completed on compatible fallback
mutation cases                     12 / 12
policy SHA-256                     d6ca2f8ebf8cf9bc2e614bbc831d92570b72b74c095a541f68fd75d965c6a80c
attempts SHA-256                   f40001089e94b30866cbb081ea08ae6a328336ada0eb19c2e1cbb2331b09de5c
receipt SHA-256                    158380fbe2907f873fd2216731e071e15fb699018dab9bd17566bd6dca5990fb
report bytes                       2,218
report SHA-256                     5f70ad4ee344102a6e9b53464b91e8c67335f10f9aa5d380455adf1ce6418f8c

Ten module tests and two example tests cover single ownership, exact compatibility, boundary control, idempotency and request joins, deterministic schedule, same-route retry, fallback, non-retryable and indeterminate outcomes, attempt/cost/route/deadline terminals, post-terminal attempt rejection, strict decoding, replay, and receipt mutation.

One logical request, many physical attempts

Use separate identities:

Identity	Meaning	Stable across retry?
run envelope	whole bounded task execution	yes
logical request	exact semantic model/tool request	yes
idempotency key	one logical operation at a receiver	yes
physical attempt	one dispatch to one route	no
route release	exact adapter/model/provider path	no, if falling back
output receipt	exact accepted result	only after success

If a new prompt, context manifest, output contract, seed policy, or required modality changes, it is not the same logical request. Create a new request identity and charge the model-call budget.

An HTTP connection retry may be a second physical attempt even when the application saw one SDK call. Hidden attempts still consume capacity, quota, and sometimes money. Either disable lower-layer retries or configure and expose them so the harness’s total bound remains true.

Retry ownership

A request may cross:

workflow → harness → provider adapter → HTTP client → load balancer → provider

If three layers each retry twice, one logical request can become:

3 × 3 × 3 = 27 physical attempts

The reference policy accepts only:

retry_owner: RetryOwner::Harness

Adapter and TransportSdk exist in the enum so a decoded policy cannot erase the distinction, but the evaluator rejects them. This reference needs the harness to observe every attempt, fallback, delay, cost, and outcome.

Another architecture may assign ownership to a durable workflow engine. Then the harness activity must disable its own retries and report one attempt faithfully. The invariant is not “the harness always retries”; it is “one layer owns the complete retry budget.”

What lower layers may still do

Connection setup can contain internal address iteration or a transport may retransmit packets. Do not classify every TCP behavior as an application attempt. The important boundary is whether the receiver may perform or bill the logical operation again.

Document:

SDK retry configuration;
HTTP connection and request retry behavior;
proxy retry rules;
queue redelivery semantics;
provider-side idempotency guarantees;
which counts appear in usage and trace receipts.

Outcome classification

The reference distinguishes three families.

Retryable failure

pub enum RetryableFailureKind {
    TimeoutBeforeResponse,
    RateLimited,
    Unavailable,
}

Retryable means policy permits another attempt if budgets, deadline, and idempotency still allow it. It does not mean “retry immediately.”

Even a timeout requires care. TimeoutBeforeResponse describes what the caller observed, not necessarily whether a provider accepted the work. Use this classification only for an operation whose duplicate dispatch is safe under the bound idempotency contract.

Non-retryable failure

pub enum NonRetryableFailureKind {
    Rejected,
    PolicyDenied,
    InvalidRequest,
    ProtocolViolation,
    InvalidOutput,
}

A different provider does not repair malformed input, policy denial, or a violated response protocol. Invalid model output belongs to bounded semantic repair, not transparent transport retry. Stopping preserves the real failure instead of shopping for a route that happens to ignore it.

Indeterminate failure

pub enum IndeterminateFailureKind {
    UnknownProviderAcceptance,
    StreamInterruptedAfterOutput,
}

Indeterminate is not retryable. The harness first reconciles:

Did the provider accept the idempotency key?
Does a request ID resolve to a terminal response?
Were usage units charged?
Is a partial stream a complete replayable prefix?
Can the exact operation be polled rather than resubmitted?

If the provider offers no reconciliation mechanism, retain the uncertainty and follow product policy. Do not relabel unknown as failure merely to enable retry.

Fallback is a compatibility proof

The fallback route contains:

pub struct FallbackRoute {
    pub route_id: String,
    pub route_release_sha256: String,
    pub capability_contract_sha256: String,
    pub behavior_class_sha256: String,
    pub data_boundary: DataBoundary,
    pub priority: u32,
    pub max_attempts: u32,
    pub supports_idempotency: bool,
}

The policy validates all routes before the first dispatch:

priorities are contiguous and deterministic;
route and release identities are unique;
capability contract equals the logical request’s contract;
behavior class equals the required behavior class;
data boundary does not exceed policy;
idempotency is supported;
per-route attempts are finite.

Capability compatibility

Compare real requirements, not marketing names:

input and output modalities;
context and output bounds;
JSON/schema/constrained-decoding support;
seed and sampling controls;
tool-call semantics;
streaming terminal and usage events;
safety/refusal representation;
regional/data-retention requirements;
response byte limits;
model/release pin strength.

A text-only fallback is not compatible with an image-grounded request. A provider that returns JSON text but lacks the required strict schema mode may be incompatible. Fail before dispatch.

Behavior compatibility

Two capable models can differ materially. A behavior class binds the evaluated requirements and release evidence allowed for this task. It should join to:

behavior specification;
output and evidence validators;
relevant held-out/safety evaluation gate;
prompt/program release or provider-specific lowering;
known limitations;
freshness/expiry.

“Supports JSON” is a capability. “Meets the incident-severity contract on required slices” is behavior evidence.

Fallback can intentionally use a weaker model only if the product contract allows a degraded behavior class and surfaces that degradation. It must not masquerade as the original class.

Data-boundary compatibility

The reference orders:

local process < private service < external provider

Policy states the maximum. A local or private primary cannot silently send evidence to an external provider during an outage. The ordering is a compact example; production boundaries also include region, tenant isolation, training retention, legal entity, encryption, and approved subprocessors.

Idempotency

Idempotency means repeating the same logical operation produces one intended logical effect, not that every response byte or latency is identical.

The policy and every attempt carry the same:

logical_request_sha256
idempotency_key_sha256

The receiver should store:

(tenant, operation, idempotency key)
  -> request digest
  -> status: in_progress | succeeded | failed_no_effect | indeterminate
  -> response/effect receipt
  -> expiry

When the same key returns:

same request digest + succeeded: return the stored result;
same digest + in progress: poll or return a retry delay;
different digest: reject key reuse;
expired record: follow an explicit retention policy, never guess.

Model generation idempotency

Many model APIs do not guarantee deduplicated generation by client key. The harness still uses one key to correlate attempts, but duplicate computation may occur. Charge and trace every physical attempt. If the provider supports asynchronous jobs or request retrieval, prefer them for reconciliation.

A seed is not an idempotency key. Provider kernels, model releases, batching, and sampling implementations may change results.

Tool effects

For externally visible writes, idempotency must extend to the target effect, not only the HTTP request. Chapter 11’s execution receipt binds effect ID, before/after state, and the same key. Fallback to a different executor requires a shared or coordinated idempotency store.

Deterministic schedule

The reference uses capped exponential backoff:

delay_n = min(base × 2^(route_attempt-1), maximum)
delay   = max(delay_n, bounded Retry-After)
not_before = completed_at + delay

The example:

attempt 1: private-primary, rate limited at 1,100 ms
Retry-After: 250 ms
attempt 2 cannot start before 1,350 ms

attempt 2: private-primary, unavailable at 1,450 ms
route attempt budget exhausted
fallback attempt cannot start before 1,650 ms

Every observed attempt must match the deterministic route, route release, ordinal, logical request, idempotency key, and earliest start. An operator cannot splice in an unapproved fallback.

Jitter

Identical exponential schedules synchronize clients and create a recovery spike. Add jitter in the live scheduler, but make it replayable:

bind a scheduler release and random seed;
record the sampled delay;
enforce the sampled value inside a policy interval;
never let jitter exceed deadline or retry-after constraints.

Common choices include full jitter over [0, cap] and decorrelated jitter. Measure fleet behavior, not just one request’s latency.

Deadline

Retries consume the original deadline. Do not reset it per attempt.

Before dispatch, budget:

remaining time
  >= queue delay
   + connection/setup allowance
   + attempt timeout
   + validation/commit allowance

If not_before >= deadline, the reference emits StopDeadline. A result completing at or after the deadline also loses to the deadline terminal. This deterministic precedence prevents late success from silently violating the caller contract.

Attempt and cost budgets

The policy combines:

maximum total attempts;
maximum per-route attempts;
maximum total cost;
absolute deadline.

The first exceeded bound terminates the sequence. The evaluator uses checked integer arithmetic. The live dispatcher should reserve the worst-case attempt cost before network I/O, then commit actual usage or retain a conservative charge when usage is unknown.

Fallback may have different prices. Route selection must not assume one “model call” has uniform cost. Record input/output/cache/media units and translate them through a versioned price schedule for reporting; keep raw provider usage too.

Streaming changes the retry boundary

Before any output is delivered, retry may be invisible to the caller if idempotency and budgets permit it. After output is delivered:

replacing the prefix can corrupt user-visible state;
consumers may already have parsed or acted on it;
usage may be partially known;
a second model may not continue coherently;
token boundaries differ across providers.

Choose a contract:

buffer the whole response, validate, then expose it;
expose tentative events and terminally commit or invalidate them;
support exact continuation from a provider-owned response ID;
stop as indeterminate and ask the caller to restart explicitly.

Do not concatenate fallback output onto a partial primary stream unless a tested protocol defines the join.

Circuit breakers, admission, and hedging

Retries handle individual failures. Circuit breakers and admission control protect the system.

Circuit breaker

A circuit breaker stops dispatch to a route after sufficient recent failure evidence. Its state is shared and time-dependent, so include breaker snapshot/generation in the routing receipt. A breaker is not a substitute for per-request budgets.

Use half-open probes with their own bounded capacity. Do not send ordinary traffic as uncontrolled health probes.

Retry budget

Limit retry traffic as a fraction of admitted primary traffic. During an outage, successful primary throughput falls; unbounded retries otherwise increase load precisely when capacity is lowest.

Hedging

Hedging sends a duplicate after a latency threshold and accepts one valid result. It can reduce tail latency but deliberately increases work. Use it only for idempotent operations with:

one hedge owner;
a hedge budget;
independent route capacity;
cancellation/late-result handling;
exact accounting for every attempt;
no externally visible effect before winner commit.

For expensive generation, a better design may be difficulty-aware routing or a latency-specific model tier rather than routine duplication.

Durable recovery

Persist:

logical_request_created
attempt_reserved
attempt_dispatched
attempt_observed
retry_decision_committed
terminal_committed

attempt_dispatched must be durable before crossing an effect boundary. Recovery verifies the event prefix, reacquires exclusive ownership, and identifies a reservation or dispatch without an outcome. That state is not “safe to retry”; it needs receiver reconciliation.

The reference evaluator checks an already observed attempt list. A production store should also fence stale workers by lease generation. A late result from an old owner may be retained as evidence but cannot overwrite a newer terminal decision.

Security implications

Boundary widening

Fallback can exfiltrate prompts, retrieved evidence, images, audio, or video to a route that was not allowed when the task began. Validate every route against tenant and data policy before admission.

Policy shopping

Attackers may induce errors to reach a more permissive provider. Every fallback must preserve safety policy, output validation, tool authorization, and behavior requirements. Non-retryable policy rejection stops the sequence.

Denial of wallet

Adversarial input can trigger slow responses, invalid outputs, and retry storms. Bound attempts, bytes, duration, repair, and cost; rate-limit by tenant and principal.

Idempotency-key isolation

Scope keys by tenant and operation. Store a digest rather than logging raw sensitive keys. Reject same-key/different-request use. Bound retention without making active operations forgettable.

Error-content injection

Provider error bodies and retry hints are untrusted observations. Parse strict bounded fields. Do not let an error message choose a fallback URL, policy, model, or delay outside the policy maximum.

Performance implications

Measure separately:

primary success latency;
per-route queue/connect/first-byte/stream/validation time;
backoff time;
attempts per logical request;
retry success rate by failure kind and ordinal;
fallback rate and success;
duplicate compute/usage;
deadline loss;
circuit state;
cost per successful logical request.

Average latency hides tail amplification. Report p50, p95, p99, and deadline miss rate. Slice by route, tenant, modality, request size, and outcome.

Retries that rarely succeed may lower availability by increasing load. Remove or narrow them based on outcome data, not intuition.

Failure investigations

One SDK call produced nine provider requests

Inspect retry settings in workflow, harness, SDK, proxy, and provider gateway. Reconstruct physical attempt IDs and billing records. Assign one owner, disable the others, and add a maximum-attempt integration test with a counting transport.

Private evidence reached an external fallback

Freeze the route. Compare task data policy, route manifest, compiled context, and fallback decision. Determine whether the external route was present at admission or dynamically injected. Revoke affected artifacts/credentials as needed, repair compatibility validation, and add a boundary-drift mutation.

Retry starts before `Retry-After`

Check parsing units, clock source, cap policy, scheduling persistence, and whether another layer retried independently. Retain the raw bounded header as evidence, but use the parsed policy decision for scheduling.

Same idempotency key performed two effects

Join both attempts to tenant, request digest, executor, and target effect. Inspect whether stores were shared across fallback routes and whether record expiry occurred mid-run. Reconcile final target truth, contain further writes, and make key acquisition atomic.

Partial output was followed by unrelated fallback text

Identify which bytes were committed to the client, the stream terminal state, and whether the fallback saw the exact prefix. Invalidate the composite result. Choose buffering, tentative events, provider continuation, or explicit restart; add a stream interruption fixture.

Fallback passes syntax but fails behavior

Compare the fallback’s evaluation release and behavior-class identity to the primary. Capability matching was insufficient. Block promotion until relevant held-out and safety slices pass.

Alternatives and trade-offs

No retry

Best for fast, user-retriable requests, dangerous effects without idempotency, and overload conditions where another attempt is unlikely to help.

Queue-based redelivery

Useful for durable asynchronous work. The queue owns delivery attempts; the worker must be idempotent and report attempt metadata. Visibility-timeout redelivery is still retry and must fit the same logical budget.

Provider asynchronous job

Submit once, then poll by response ID. This often handles long inference and disconnections better than resubmission. Polling needs its own rate and deadline bounds.

Active-active routing

Route each new logical request across healthy providers rather than retrying one request across many. This reduces failure concentration but requires continuously evaluated compatibility and stable traffic allocation.

User-visible retry

Stop and let the user choose. This creates a new logical request unless the protocol intentionally reuses the old idempotency identity. It is honest when fallback would change privacy, quality, or cost.

Runnable laboratory

The reference sequence is:

private-primary attempt 1
  -> rate limited, Retry-After 250 ms
private-primary attempt 2 at the checked time
  -> unavailable, route attempt budget exhausted
external-fallback attempt 1
  -> validated success

The external route is allowed only because the test policy explicitly permits it and both routes bind the same capability and behavior identities.

Twelve cases cover valid success, opaque retry ownership, contract drift, boundary drift, logical request drift, premature fallback, non-retryable failure, indeterminate outcome, total attempt budget, cost budget, deadline, and route exhaustion. The report is written to:

target/labs/harness-14/<policy-sha256>.json

Exercises

1. Recall

For each event, decide whether it is the same logical request or a new one:

reconnecting before any request bytes were accepted;
resending the same body and idempotency key;
adding validation feedback to the prompt;
changing only the route release;
changing the output schema;
a user pressing “try again” after a terminal failure.

2. Calculate

With base backoff 200 ms, maximum 1,500 ms, and no retry hint, calculate delays for route attempts 1–5. Then apply a bounded 1,200 ms retry hint to attempt 2.

3. Implement

Add replayable full jitter. Bind a scheduler release and seed, record the sampled delay, validate its range, and add same-seed replay plus out-of-range mutation tests.

4. Compatibility

Add a third route that lacks image input but supports text JSON. Prove it is rejected for a screenshot-grounded task before any primary attempt occurs.

5. Durable recovery

Persist a dispatch whose process crashes before recording the response. Implement a fake receiver that can reconcile by idempotency key. Prove recovery polls rather than resubmits.

6. Streaming

Design a tentative-event protocol for partial model output. Name the event identity, validation state, terminal commit, invalidation behavior, and client resumption rule.

7. Failure investigation

Inject retries at both SDK and harness layers. Capture physical request IDs, quantify amplification, remove one owner, and retain a regression that proves the total bound.

8. Design review

Choose between no retry, same-route retry, fallback, asynchronous polling, hedging, and user-visible retry for a high-cost video generation request. Defend quality, deadline, cost, privacy, and cancellation behavior.

Solution guidance

A semantic prompt/schema change creates a new request. Route change can retain the logical request only if compatibility holds. User retry is normally new unless the product resumes the old operation explicitly.
Exponential values are 200, 400, 800, 1,500, 1,500 ms. A 1,200 ms hint raises the second delay to 1,200 ms.
Use deterministic pseudorandom input, persist the chosen delay, and reject values beyond cap, hint, or deadline rules.
Modality is part of the capability contract. Route-policy validation should fail without looking at runtime health.
Persist dispatch before I/O, then query the receiver. Only verified no-effect permits a new dispatch.
Clients must distinguish tentative from committed output and handle explicit invalidation.
One logical call should expose every physical receiver attempt in a counting fixture.
Expensive generation often favors submit-once/poll and explicit user decisions over opaque cross-provider fallback.

Check your understanding

Why can independent retries at two layers multiply rather than add?
Why is a seed not an idempotency key?
Which failures must not fall back?
What does capability compatibility omit that behavior compatibility supplies?
Why does fallback need a data-boundary proof?
How should a retry hint interact with exponential backoff?
Why must the deadline remain original across attempts?
What makes a streamed interruption indeterminate?
How does circuit breaking differ from retry?
What must a durable worker do with an unmatched dispatch event?

Practical completion checklist

Authoritative references

IETF, RFC 9110 · HTTP Semantics: safe/idempotent method semantics, request expectations, and status behavior.
IETF, RFC 6585 · Additional HTTP Status Codes: the 429 Too Many Requests status and optional Retry-After.
Amazon Builders’ Library, “Timeouts, retries, and backoff with jitter”: load amplification, timeout selection, capped backoff, jitter, and retry ownership.
Google SRE Book, “Handling Overload”: admission, load shedding, retry behavior, and protecting useful throughput.
Kleppmann, “Designing Data-Intensive Applications,” Chapter 11: stream processing, idempotence, derived state, and end-to-end reasoning about repeated delivery.
NIST, AI Risk Management Framework 1.0: lifecycle risk, measurement, monitoring, and governance context for route changes.