Authentication, Authorization, Tenancy, and Quotas

Authentication answers “which principal presented valid proof?” Authorization answers “may this principal perform this action on this object now?” Tenancy answers “which isolation domain owns the object and spend?” Quota answers “how much scarce resource may that domain reserve and consume?”

Combining them into one is_admin boolean is a security and operations failure.

The companion mosaic-policy crate begins after cryptographic authentication. It provides validated principal/tenant identities, explicit capabilities, fail-closed object tenant checks, and a reservation-based quota ledger with concurrent and unit bounds.

Learning objectives

• separate authentication, authorization, tenancy, and quota;
• validate bearer/JWT/OIDC claims with explicit trust configuration;
• inject verified identity rather than trusting body/path tenant fields;
• authorize action and object ownership on every access;
• model capabilities instead of ambient roles/booleans;
• reserve estimated cost/concurrency atomically before work;
• commit actual usage or refund reservations safely;
• distinguish 401, 403, 404, 429, and 503;
• test negative cross-tenant, missing-capability, race, and resource-exhaustion cases.

Dependencies and mental model

Depends on Axum middleware/extractors, Tower order, SQLx transactions, durable jobs, and artifact storage.

credential/token/mTLS proof
        │ cryptographic + issuer/audience/time validation
        ▼
verified principal claims
        │ map through current server policy
        ▼
Principal { subject, tenant, capabilities }
        │
        ├── authorize action + object tenant/state/classification
        ├── reserve quota/capacity
        └── execute narrow capability

Unverified token claims are untrusted input. A signed token is also not automatically authorized: signature validity does not prove correct issuer, audience, token type, time, scope, revocation policy, or object ownership.

Authentication choices

OAuth/OIDC access tokens

Common for user/service APIs. The resource server validates access tokens for its audience. An OIDC ID token identifies an authentication event/client session; it is not automatically the right credential for an arbitrary API.

JWT validation must configure:

• exact issuer;
• exact intended audience/resource;
• allowlisted algorithm/key use;
• signature and key selection;
• expiration/not-before with bounded skew;
• token type/profile;
• subject/client identity;
• required scopes/claims;
• maximum token/header size;
• JWKS retrieval/cache/rotation/failure policy;
• mutually exclusive rules for different token kinds.

Do not decode a JWT and trust its JSON. Do not select algorithms from the attacker without an allowlist. Do not accept a token issued for another service.

Opaque token introspection centralizes revocation/current state but adds a network dependency, cache/staleness, latency, and availability policy.

Service identity/mTLS/workload identity

Useful between workloads. Verify chain/SPIFFE or platform identity, expected trust domain/audience, and bind to least-privilege service permissions. “Inside the cluster” is not authentication.

API keys

Appropriate for some machine integrations:

• high-entropy generated secret;
• public key ID plus secret verifier;
• store a slow/appropriate one-way verifier or keyed hash under threat model;
• show secret once, never log;
• scope, expiry, rotation, last-used/audit;
• constant-time verification where needed;
• rate/tenant policy;
• no key in query string.

Sessions/cookies

Browser applications need secure/HTTP-only/same-site cookie policy, CSRF defense, origin/CORS design, rotation, session fixation prevention, and logout/revocation semantics.

Middleware placement

One safe high-level order:

edge connection/header limits
→ request identity/tracing shell
→ strict body-size guard
→ credential extraction/authentication
→ principal capability derivation
→ authorization/quota
→ body parse/domain validation
→ durable transaction

If a signed request authenticates body bytes, stream/hash under a strict limit before verification. Do not buffer unlimited input first.

Authentication middleware attaches a typed Principal extension/state. Handlers do not reread raw authorization headers or accept tenant_id from the body as identity.

The companion principal and capabilities

pub struct Principal {
    pub subject: PrincipalId,
    pub tenant: TenantId,
    capabilities: BTreeSet<Capability>,
}

pub enum Capability {
    RunsCreate,
    RunsRead,
    RunsCancel,
    ArtifactsRead,
    ArtifactsWrite,
    PolicyAdmin,
}

PrincipalId and TenantId reject empty, oversized, control-character values, including through Serde. They are opaque identifiers—not paths or log-safe labels by default.

Capabilities express actions. A role/group/token scope is an input to current server policy that derives capabilities. Do not let a stale long-lived token permanently carry every mutable authorization decision when revocation/entitlement changes must be immediate.

The companion intentionally has no implicit “admin may do everything.” Powerful operations need explicit capabilities and often step-up/multi-party controls.

Object-level authorization

Every endpoint taking an object ID checks:

authorize(&principal, &target_tenant, Capability::RunsRead)?;

The companion returns CrossTenant before capability success if object tenant differs. Production repositories should make bypass difficult:

load_run(tenant_id, run_id)
cancel_run(tenant_id, run_id, expected_version)
load_artifact(tenant_id, artifact_id)

and SQL:

SELECT ...
FROM runs
WHERE tenant_id = $1 AND run_id = $2;

Do not load globally by run_id, then remember to check tenant later. Include tenant in composite unique/foreign keys and, where appropriate, database row-level security as defense in depth.

Authorization includes:

• principal/tenant/action;
• object ownership/classification/state;
• relationship/team/project;
• time/device/network/risk policy;
• data residency/provider/tool allowance;
• approval/step-up;
• field/property-level projection.

Creation needs authorization too: caller must not choose a different tenant, privileged risk class, model route, tool capability, retention, or destination through mass assignment.

403 versus existence hiding

• 401: missing/invalid authentication; include correct WWW-Authenticate policy.
• 403: authenticated but forbidden.
• 404: may hide object existence when policy requires and behavior is consistent.
• 429: principal/tenant rate or quota exhausted.
• 503: global/provider/service capacity unavailable.

Existence hiding must include timing/error/body/header/cache behavior where threat model requires it. Operators still need internal reason codes. Do not use 404 to avoid implementing authorization.

Quotas are correctness and cost control

AI quotas include:

• concurrent runs/provider calls/GPU jobs;
• input bytes/pixels/audio seconds/video frames;
• tokens/embedding items/generated images/video seconds;
• tool calls/browser/subprocess time;
• storage/event/trace bytes;
• cost/currency units;
• requests over time.

Enforce at the earliest trustworthy point and again at actual consumption. A request-count rate limit cannot stop one request from spending a large model/media budget.

Reservation before execution

Estimated work must reserve capacity:

charged_units + reserved_units + requested_units <= window_limit
active_runs + 1 <= concurrent_limit
input_bytes <= per-run limit

The companion:

let reservation = ledger.reserve_run(
    &principal,
    &tenant,
    input_bytes,
    estimated_units,
)?;

// durable admission and work

reservation.commit(actual_units)?;

On abort/drop, it releases active count and reserved units. On commit, it charges actual units and releases unused reservation. Actual usage above the reservation fails instead of silently exceeding policy.

This prevents many concurrent requests from each observing the same remaining quota and oversubscribing it.

Reservation implementation

QuotaLedger protects one policy-window map with a mutex:

struct QuotaUsage {
    active_runs: u32,
    reserved_units: u64,
    charged_units: u64,
}

Reservation mutation happens in one critical section with checked arithmetic. QuotaReservation owns the reservation; Drop is a best-effort safety net, while explicit commit/abort returns errors.

Six tests prove:

• cross-tenant and missing capability fail closed;
• dropped/aborted reservation refunds;
• commit charges actual and releases unused estimate;
• input and actual bounds;
• four racing threads yield exactly two reservations under limit two;
• malformed identities cannot enter through deserialization.

This is correct only inside one process/window instance. Multi-replica authoritative quota must be a transactional database/strong counter service, or use deliberately conservative local allocations from a global budget.

Durable quota transaction

Run admission should atomically:

1. authenticate outside transaction;
2. load/lock current tenant quota window;
3. authorize current entitlement;
4. validate input/artifact usage;
5. reserve run/cost units;
6. create idempotency/run/event/job;
7. commit;
8. return 202.

Unique/idempotency ensures retries do not reserve twice. Worker completion atomically:

• records authoritative usage;
• converts/reconciles reservation to charge;
• releases concurrency;
• updates run/event.

Crash recovery needs reservation expiry tied to job state, but do not expire while a valid leased job still owns it. A reconciler detects terminal runs with outstanding reservations and vice versa.

Estimates, actual usage, and overruns

Estimate from bounded input/model/output/tool/media plan. Add safety margin without reserving the entire tenant budget unnecessarily.

If actual approaches reservation:

• stop at a deterministic safe boundary;
• request incremental reservation transactionally;
• route to a cheaper bounded plan if policy permits and trace it;
• return partial result only under explicit contract;
• never continue unlimited spend and “bill later.”

Usage source must be defined. Provider-reported tokens, local tokenizer counts, video duration, and storage bytes can differ. Record source/version and reconcile.

Fairness and hierarchical capacity

Hierarchy:

global service/GPU/provider capacity
  └── tenant allocation/quota
       └── principal/project
            └── run/step budget

Acquire in consistent order or use one admission scheduler to avoid deadlock. Global 503 and tenant 429 are distinct. One tenant must not fill global in-memory queues before tenant quota is checked.

Fair scheduling can use weighted queues, reserved floors, burst ceilings, and priority classes. Priority is policy input, not arbitrary client integer.

Token/key rotation and cache failure

JWKS/policy caches need:

• bounded TTL/stale policy;
• key ID miss refresh without attacker-driven fetch storms;
• issuer-pinned URLs (no SSRF);
• last-known-good behavior bounded by revocation requirements;
• rotation overlap;
• metrics without tokens/subjects;
• startup/readiness decision.

Fail-open versus fail-closed is a documented risk choice per operation. Creation/cancel/admin should normally fail closed when authorization state cannot be established.

Audit without surveillance leakage

Audit decisions:

• principal/service stable internal ID;
• tenant/action/object type/opaque ID;
• allow/deny reason code/policy revision;
• request/run/trace ID;
• quota reservation/charge class;
• timestamp/outcome.

Do not log bearer/API keys, raw claims, prompts/media, sensitive object names, signed URLs, or unbounded policy documents. Audit access and retention are themselves authorized.

Runnable companion implementation

cd rust/rust-ai-engineering
cargo run -q -p mosaic-policy --example tenant_quota
cargo test -p mosaic-policy

Expected:

tenant=tenant-demo active=0 reserved=0 charged=37 window=2026-07-28

The example reserves 60 units, commits actual usage 37, releases concurrency, and shows no dangling reservation.

Failure investigation: user reads another tenant’s run

1. preserve request/identity/policy/repository evidence;
2. identify every endpoint/query that accepts object ID;
3. check whether tenant came from verified principal or request input;
4. inspect repository query predicates and cache keys;
5. revoke/contain and assess exposed fields/artifacts/events;
6. structurally add tenant to repository/API keys and database constraints;
7. build two-tenant negative matrix for read/update/delete/stream/download.

Do not patch only the reported route.

Failure investigation: quota shows negative or exceeds limit

1. reconcile active/reserved/charged with runs/jobs/usage events;
2. inspect non-atomic check-then-increment paths;
3. check retries/idempotency double reservation/charge;
4. inspect drop/abort/terminal/recovery paths;
5. verify integer units/window/time boundary;
6. freeze unsafe admission and repair under audited transaction;
7. add synchronized concurrency/failure tests and invariants.

Failure investigation: valid tokens intermittently fail after rotation

1. inspect issuer/audience/algorithm/time and safe error class;
2. compare key IDs and JWKS cache versions across replicas;
3. check refresh concurrency/rate/timeout/DNS/TLS;
4. confirm old/new key overlap and clock skew;
5. ensure unknown kid cannot trigger unbounded external fetch;
6. rehearse rotation with deterministic fixtures and multiple replicas.

Alternatives and trade-offs

Self-contained JWT

Low-latency local validation and distributed operation. Revocation/current entitlement/key cache and claim staleness require policy.

Opaque introspection/session

Central current decisions/revocation. Adds network dependency/cache/latency.

Role-based access

Simple grouping. Roles become coarse/ambient. Map roles to explicit action/object capabilities.

External policy engine

Centralized rich policy/audit. Adds service/latency/version/cache/failure concerns; application must still pass correct attributes and enforce result.

In-process quota

Fast and testable. Not authoritative across replicas/restarts. Useful as a local lower bound in front of durable quota.

Security and performance implications

Security:

• cryptographic validation uses maintained libraries and pinned trust rules;
• bearer tokens use TLS, short life/scope, never URL/log;
• tenant derives from verified identity/current policy;
• every object/property/action is authorized;
• database/cache/object keys include tenant;
• quotas precede expensive parse/decode/inference;
• capability/entitlement/policy revisions are traced;
• deny/default/failure mode is explicit.

Performance:

• token verification/JWKS/policy lookup needs bounded caches;
• high-cardinality subjects/tenants are not metric labels;
• centralized quota row can become hot; shard/window/bucket carefully;
• strict transaction locks protect correctness but add contention;
• local reservations reduce global calls only with conservative allocation;
• per-media measurement should stream, not buffer;
• authorization/query must use indexes including tenant.

Exercises

1. Add an Axum Principal extractor backed by a verified-test authenticator trait; no raw claims in handlers.
2. Add tenant to mosaic-db run/job/event/idempotency schemas and every query.
3. Persist quota reservation with durable run admission in one transaction.
4. Design JWT validation configuration and negative fixtures for issuer/audience/algorithm/time/type.
5. Create a two-principal/two-tenant authorization matrix for every API route.
6. Add incremental quota reservation when actual model usage approaches estimate.
7. Design key/policy cache rotation and fail-closed behavior during issuer outage.

Solution guidance

Authenticator returns only a validated Principal; production crypto remains in a maintained OIDC/OAuth library or trusted gateway with authenticated context. Repository APIs require tenant arguments, and composite foreign keys prevent cross-tenant references. Durable reservation uses a locked quota-window row plus idempotent run transaction. Negative fixtures include wrong issuer, audience, algorithm, type, expired/not-yet-valid, unknown key, duplicate claim, and oversized token.

Check your understanding

• Why is a valid JWT not automatically authorized?
• Where does tenant identity come from?
• What is object-level authorization?
• Why reserve estimated units before work?
• How do drop, abort, and commit differ?
• What does the in-process ledger fail to protect?
• When should a response be 429 versus 503?
• Why should admin not imply every capability automatically?

Completion checklist

• Authentication trust rules validate issuer/audience/algorithm/time/type.
• Raw unverified claims never become Principal.
• Tenant comes from verified/current server policy.
• Every route/repository/object/property action is authorized.
• IDs/cache/database/object keys are tenant scoped.
• Quota reservation and durable admission are atomic.
• Actual usage commits and unused reservation releases.
• Retry/idempotency cannot double reserve/charge.
• Concurrent/multi-replica quota is authoritative.
• Negative auth/tenant/quota tests dominate the suite.
• I ran the policy example and six tests.

Authoritative references

• RFC 8725: JWT Best Current Practices
• RFC 6750: OAuth bearer token usage
• OpenID Connect Core 1.0
• OWASP API Security Top 10 2023
• OWASP Authorization Cheat Sheet
• OWASP REST Security Cheat Sheet