Containers, Health, Shutdown, Rollout, and Rollback

A deploy is a distributed state transition, not a file copy:

reviewed source
  → one tested release binary
  → one content-addressed image
  → staged replicas with mixed old/new versions
  → traffic and durable work
  → promote, pause, or roll back using predefined evidence

The hardest bugs live between those arrows:

a pod reports healthy before migrations/model load/worker ownership;
liveness kills an overloaded process and amplifies the overload;
SIGTERM stops HTTP but abandons accepted jobs;
old and new binaries disagree about database, events, prompts or model artifacts;
a floating image tag changes between environments;
rollback restores code but cannot reverse destructive data/model changes;
a canary looks green because it received only easy text requests, not long video work.

The companion now contains a deployable mosaic-service binary, three distinct health endpoints, signal-driven admission draining, a bounded worker drain, a multi-stage digest-pinned Dockerfile, and a hardened Kubernetes Deployment/Service/PodDisruptionBudget template. A real socket and container smoke test exercise the lifecycle.

This milestone is honest about one remaining boundary: until later book sections connect a model provider, the lifecycle worker records model_provider_not_configured as a terminal failure. It never fabricates AI success.

Learning objectives

distinguish process, container, pod, service and durable-job lifecycles;
build one optimized binary into one minimal, non-root, content-addressed image;
pin and deliberately update builder/runtime/frontend images;
understand what lockfiles, SBOMs, provenance, signatures and scans do and do not prove;
implement separate startup, readiness and liveness semantics;
prevent traffic admission before startup and during draining;
handle SIGTERM/control-C with truthful ordering and a bounded deadline;
preserve accepted durable work across process death and fence stale workers;
configure Kubernetes rolling updates, disruption budgets and security context;
design expand/migrate/contract compatibility across mixed versions;
stage by modality, model/harness/schema and infrastructure dimensions;
define automated/manual promotion and rollback gates before release;
verify rollback restores service without corrupting or misinterpreting state.

Dependencies and mental model

This chapter depends on:

deterministic build/dependency gates;
configuration validation and secret injection;
readiness of database/object/model dependencies;
bounded admission and backpressure;
durable jobs, leases, idempotency and recovery;
structured cancellation;
health-safe observability and benchmark/capacity evidence.

The process state machine is:

constructing
  → bound + initialized + worker owns queue
  → ready / accepting
  → signal or fatal dependency policy
  → draining / not accepting
  → connections complete + senders drop
  → queued work drains or lease is released/expires
  → telemetry flush
  → exit

Crashes, node loss and SIGKILL can skip every graceful step. Correctness therefore comes from durable state, idempotency and fencing. Graceful shutdown reduces interruption; it is not the only recovery mechanism.

Step 1: define the release unit

Promote the same image digest from test to staging to production. Rebuilding from the same Git revision creates a new candidate unless byte-for-byte reproducibility is proven.

A release manifest includes:

source revision and dirty status;
Rust/Cargo/target/profile/features;
lockfile and build-script/native inputs;
builder/runtime base image digests;
application image manifest digest and per-platform manifests;
SBOM, dependency/advisory/license policy;
build provenance/attestation and signatures;
test/fuzz/benchmark/eval/security report IDs;
database/event/config/API compatibility range;
model/tokenizer/preprocessor/harness/prompt/tool/policy/index artifacts;
deployment/config/feature-flag revision;
owner, change summary and rollback decision.

An image digest identifies bytes/metadata, not benevolence. A signed vulnerable image is authentically vulnerable. Verification policy must bind an approved identity/workflow/source and still evaluate content.

Step 2: build a small pinned container

The companion Dockerfile uses a build stage and a separate runtime stage:

# syntax=docker/dockerfile:1@sha256:<reviewed-frontend-digest>

FROM rust:1.96.0-bookworm@sha256:<reviewed-builder-digest> AS builder
WORKDIR /workspace
COPY Cargo.toml Cargo.lock rust-toolchain.toml rustfmt.toml ./
COPY crates ./crates
RUN cargo build --locked --release --package mosaic-api --bin mosaic-service

FROM debian:bookworm-slim@sha256:<reviewed-runtime-digest> AS runtime
COPY --from=builder \
  /workspace/target/release/mosaic-service \
  /usr/local/bin/mosaic-service
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/mosaic-service"]

The checked-in file contains the full digests resolved and verified on July 28, 2026. A reviewed dependency update must resolve new digests, build, scan, test, and record why it moved. Digest pinning intentionally opts out of invisible base-tag updates, so update automation and security response are mandatory.

containers/.dockerignore excludes target artifacts, fuzz workspace, Git metadata and Markdown. That reduces context size and accidental leakage. It does not replace a secret scan.

Build:

cd rust/rust-ai-engineering
docker build \
  --file containers/Dockerfile \
  --tag mosaic-service:lifecycle \
  .

The verified local build produced an OCI image manifest list identified by:

sha256:78fc902b0fb9aa9b3ec5f440c2fc5784ba4b509269c165e31a2e1a1a74f05dff

That is local evidence for this source state, not a published production digest.

Why copy manifests and crates together?

This first Dockerfile favors clarity and a small build context. Copying all crate sources before cargo build invalidates dependency compilation when source changes. Production BuildKit can use cache mounts or a dependency-planning stage, but:

cache is an optimization, not release evidence;
cache keys include target/toolchain/lock/features/build environment;
untrusted builds must not poison trusted executable caches;
final binary must still be built from the reviewed source in the attested job.

Runtime contents

The runtime image contains Debian’s minimal base and one binary; it has no Rust compiler or source. The current lifecycle service makes no outbound TLS request. When a later remote provider/database adapter requires CA roots or native libraries, add the exact runtime dependencies deliberately and test them. “Slim” is not “distroless,” vulnerability-free or complete.

The numeric non-root user works without a writable home or password entry. The service binds 8080, so it does not need privileged-port capability.

Build secrets

Never put tokens in ARG, ENV, copied .env, registry URL, build log or layer. Build arguments can appear in provenance. Use BuildKit secret/SSH mounts only when unavoidable, scope credentials, and ensure no generated output/layer retains them. Prefer dependency inputs accessible without secrets.

Multi-platform images

An OCI index can point to architecture-specific manifests. Test each supported native target:

binary/native library availability;
model runtime and accelerator provider;
numeric/quality tolerance;
image size/startup/performance;
seccomp/syscall behavior;
SBOM/provenance/signature coverage.

Emulation proves less than a native run, especially for SIMD/accelerators.

Step 3: harden runtime defaults

The Kubernetes template and verified docker run use:

UID:GID 65532:65532
read-only root filesystem
all Linux capabilities dropped
no-new-privileges
RuntimeDefault seccomp
no service-account token mounted
explicit CPU/memory requests and memory limit

The local runtime test:

docker run --detach \
  --name mosaic-service-lifecycle-test \
  --read-only \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --publish 127.0.0.1:18081:8080 \
  mosaic-service:lifecycle

Inspection confirmed the numeric non-root user and all three host restrictions while readiness returned {"status":"ready"}.

Containers share the host kernel. These settings reduce privileges; they do not create a perfect sandbox. Add network policy, workload identity, node isolation/sandbox runtimes, AppArmor/SELinux and separate GPU pools according to threat model.

Write only to explicitly mounted, quota/size-limited volumes. A read-only root filesystem catches accidental local persistence. Durable runs/artifacts belong in database/object storage, not a pod filesystem.

Set a memory limit only after measuring workload peaks. In many environments, exceeding it leads to an abrupt OOM kill with no graceful drain. Queue/input/batch/media/model memory bounds must keep the process below it with headroom.

Step 4: health endpoints state facts, not optimism

The companion exposes:

GET /health/startup
GET /health/ready
GET /health/live

Startup

Answers:

Has this process completed required one-time initialization so ordinary liveness/readiness checks may begin?

The binary marks startup complete only after:

environment configuration parsed;
API/queue constructed;
TCP listener bound;
worker task owns the receiver.

Future local-model service also waits for validated model/tokenizer/preprocessor load and a bounded smoke inference if that is required to serve.

Startup failure should keep the pod from receiving traffic and eventually cause restart according to probe policy. Budget enough time for measured cold load. Do not give a permanent failing process hours to hide.

Readiness

Answers:

Should this instance receive new traffic now?

It is false:

before startup completes;
after drain begins;
when admission cannot safely accept;
for a critical dependency state that makes this replica unable to serve its contract.

It should not flap on every transient downstream request failure. Readiness removal shifts load to other replicas; if all replicas share the dependency, removing all can turn degradation into outage. Use circuit breakers/explicit errors and decide dependency policy.

The companion admission handler consults the same ServiceHealth state. Starting/draining requests receive stable 503 service_not_ready; the health endpoint and actual behavior cannot disagree by configuration alone.

Liveness

Answers:

Is this process irrecoverably unable to make progress, such that restart is safer?

The companion liveness is a cheap event-loop/router response and does not query the model provider, database or internet. Restarting a healthy process during provider slowness adds cold starts and load. A deeper watchdog may detect a deadlocked executor, but it must be designed/tested against real failure modes.

Health response safety

Health endpoints are small, fast, unauthenticated within their intended network and reveal minimal status. They do not expose secrets, dependency URLs, model paths, versions useful to attackers, or full error chains. Restrict their network exposure even if content is minimal.

Step 5: make state transitions atomic and test them

ServiceHealth owns two atomics:

startup_complete: AtomicBool,
accepting: AtomicBool,

Its visible phases:

pub enum ServicePhase {
    Starting,
    Ready,
    Draining,
}

mark_ready enables acceptance then publishes startup completion. start_draining disables acceptance and keeps startup complete. Acquire/release ordering communicates the lifecycle flags; it does not make every concurrent admission instantaneously disappear. A request already beyond the admission check may still enter the bounded queue. Shutdown therefore drains the queue.

Tests prove:

startup/readiness return 503 initially;
admission returns service_not_ready;
after mark_ready, readiness is 200 and work queues;
after start_draining, readiness/admission return 503;
startup completion never returns to false.

State should be monotonic inside one process. A temporarily unhealthy dependency can use a distinct not-ready state and recover; a shutdown drain should not spontaneously become accepting again.

Step 6: order graceful shutdown correctly

The binary waits for control-C or SIGTERM. On signal:

1. mark not ready / reject new admission
2. tell Axum to stop accepting and drain connections
3. drop router-owned queue senders
4. worker receives remaining accepted items, then channel closes
5. await worker under MOSAIC_DRAIN_SECONDS
6. abort + join if deadline expires
7. emit final structured report
8. exit

The relevant code:

health.start_draining();

let server_result = axum::serve(listener, api.router)
    .with_graceful_shutdown(shutdown_signal(health))
    .await;

let drain_result = finish_worker(worker, drain_budget).await;

In the actual binary the signal future flips health before it resolves and triggers Axum’s graceful path.

The container test received SIGTERM and emitted, in order:

mosaic.service.ready
mosaic.service.draining
mosaic.service.stopped

The final report showed zero lost/error events for the empty smoke run.

The deadline hierarchy

Configure:

request/stream drain
  + worker release/checkpoint
  + telemetry flush
  + safety margin
  < orchestrator terminationGracePeriodSeconds

The template gives the process a 20-second worker budget inside a 30-second pod grace period. Axum connection drain currently shares the process time before worker join; long-lived SSE clients can consume that window. A production policy should stop/notify streams, bound connection drain separately, then preserve time for jobs/telemetry.

Kubernetes usually sends TERM and force-kills remaining processes after the grace period. SIGKILL, OOM, node loss and power loss give no cleanup opportunity.

Durable versus in-memory work

The current HTTP milestone uses an in-memory bounded queue for local demonstration. Production accepted work must be created transactionally in the durable jobs table before returning 202. Then shutdown can:

stop claims;
complete a short safe unit if time allows;
checkpoint only where semantics support it;
release the lease or let it expire;
let a new generation recover;
fence late completion from the old worker.

Never mark a job failed merely because one pod is terminating unless that is the logical terminal result. Never rely on graceful shutdown to preserve the only copy of accepted work.

Side effects during forced drain

An external provider/tool may have succeeded while the worker loses its response. Cancellation cannot undo it. Persist a stable operation key/receipt, reconcile before retry, and make side-effect idempotency separate from database job uniqueness.

Step 7: understand pod termination

When a pod is marked terminating, current Kubernetes EndpointSlice behavior marks the endpoint non-ready for ordinary traffic while kubelet begins local shutdown and sends TERM. Control-plane, load-balancer, connection and signal actions are concurrent, not a globally ordered transaction.

Therefore:

application flips readiness as soon as it sees shutdown;
clients retry only safe/idempotent operations;
connection keep-alive/stream policy is explicit;
accepted work is durable;
grace includes propagation delay/headroom;
test with the real ingress/service mesh/runtime.

A blind preStop: sleep 10 spends the same grace period and guesses propagation time. Use it only for a proven platform integration need; application readiness and durable semantics remain necessary.

Step 8: the Kubernetes rollout template

The checked-in template specifies:

replicas: 3
minReadySeconds: 10
progressDeadlineSeconds: 300
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

maxUnavailable: 0 maintains desired ready capacity during ordinary rollout when the cluster can schedule the surge. maxSurge: 1 adds one candidate at a time. This costs extra capacity and is not a guarantee against node/zone failures, bad readiness, quota exhaustion or correlated defects.

minReadySeconds requires a new pod to remain ready before availability. progressDeadlineSeconds makes stalled progress visible; Kubernetes reports it but does not autonomously choose the business rollback.

The PodDisruptionBudget keeps two of three replicas available for voluntary disruptions. A PDB:

does not stop all involuntary failures;
does not fix bad application rollout strategy;
can block maintenance if too strict;
must account for autoscaling and topology.

Image marker

The manifest contains:

image: REPLACE_WITH_IMMUTABLE_IMAGE_DIGEST

Release rendering must replace it with:

registry.example/mosaic-service@sha256:<approved-application-digest>

Do not apply the template directly. CI/policy should reject non-digest production images and the marker. imagePullPolicy is not a substitute for digest identity.

Probes

startupProbe:
  httpGet: { path: /health/startup, port: http }
  periodSeconds: 2
  failureThreshold: 60

readinessProbe:
  httpGet: { path: /health/ready, port: http }
  periodSeconds: 5
  failureThreshold: 2

livenessProbe:
  httpGet: { path: /health/live, port: http }
  periodSeconds: 10
  failureThreshold: 3

While startup probe has not succeeded, Kubernetes defers readiness/liveness. Tune thresholds from measured cold starts and failure detection/restart costs. One-second timeouts require the health path to be isolated and cheap under load.

Resources and scheduling

Requests influence scheduling and autoscaling utilization; limits constrain runtime. The template’s CPU/memory values are teaching defaults, not capacity evidence. Replace them using the performance chapter and load/soak tests.

For accelerator workloads also model:

device resource requests and compatible node labels;
model weights/KV/cache/batch memory;
topology/NUMA/PCIe transfer;
one model per replica versus shared inference service;
long startup and image/model download;
local cache versus content-addressed remote artifact;
surge feasibility when every old replica already occupies a GPU;
preemption/checkpoint/recovery.

maxSurge: 1 cannot work if there is no spare accelerator. Blue/green, external model service, temporary capacity or a controlled unavailable replica may be required.

Step 9: schema and mixed-version compatibility

During a rolling deployment, old and new versions run simultaneously. They may read/write the same:

database rows and migrations;
event/trace JSON;
object manifests;
queue messages;
API/SSE contracts;
configuration;
cache/index;
model/harness outputs.

Use expand/migrate/contract:

expand: add nullable/new field/table/event variant that old code safely ignores;
deploy code that reads old/new and writes a compatible form;
backfill with resumable, bounded, observable jobs;
switch reads/writes under a reversible control;
prove no old producer/consumer remains;
contract: remove old form in a later release.

Destructive migration and code release in one step can make rollback impossible. Database rollback usually means roll forward with a corrective migration, not blindly reversing after new writes.

AI compatibility is schema compatibility

A prompt/tool/model change can alter JSON fields, stop reasons, tool selection, embeddings or media artifacts while the Rust API remains identical. Version:

prompt/harness/tool schema;
model/tokenizer/preprocessor/quantization;
retrieval embedding/index;
output contract/evaluator;
safety/policy;
artifact manifest.

During transition, route a coherent bundle. Never query an index with a different embedding space without an explicit compatibility layer.

Step 10: staged rollout

A robust sequence:

freeze release manifest and artifact digest;
verify signature/provenance/SBOM/policy;
run deterministic, integration, fuzz, benchmark and eval gates;
test migration/restore/rollback in an isolated production-like environment;
deploy to internal/dev with synthetic traffic;
shadow safely when side effects/content policy permit;
canary a small but representative population;
hold for enough traffic/time to observe delayed failures;
increase stages while comparing candidate versus control;
complete rollout, then watch a post-deploy window;
contract schemas only in a later release.

Canary assignment should be stable by tenant/run and prevent one logical run from switching harness mid-flight. Exclude neither hard nor expensive cases unintentionally. Slice by:

task/modality/input-size/duration/language/risk;
tenant tier/region/provider/model;
cached versus cold/local versus remote;
repair/tool/escalation paths;
software and model/harness/schema revision.

Do not send sensitive production content to an unapproved shadow backend. Do not duplicate side-effecting tools/provider charges in shadow.

Promotion criteria

Predefine:

rollout health and ready replica count;
error/rejection/cancellation/timeout rates;
queue age/depth and lease recovery;
p50/p95/p99 and first-output latency;
CPU/RAM/device/storage/network saturation;
telemetry drop/export failures;
task quality/eval/safety by slice;
tool/repair/escalation amplification;
cost per successful run;
security/integrity signals;
minimum sample/time and uncertainty policy.

Absence of alerts is not promotion evidence if the candidate processed too little traffic or telemetry failed.

Step 11: rollback is a designed operation

Rollback trigger examples:

SLO burn/error spike;
crash/OOM/restart or readiness failure;
queue age/recovery surge;
latency/cost/repair regression;
quality/safety gate breach;
corruption, cross-tenant or security signal;
incompatible consumer/schema behavior.

Immediate containment may be:

pause rollout;
route candidate to zero;
disable a feature/tool/model with reviewed kill switch;
reduce input/batch/concurrency;
fail over provider/model;
revoke a credential;
stop affected durable claims.

Then restore the last known-good digest and coherent configuration/model/schema bundle. A Kubernetes Deployment revision rolls back the Pod template, not database/object writes or external side effects.

Commands in a controlled environment:

kubectl rollout pause deployment/mosaic-service
kubectl rollout history deployment/mosaic-service
kubectl rollout undo deployment/mosaic-service --to-revision=<reviewed>
kubectl rollout status deployment/mosaic-service

Prefer deployment GitOps/release tooling that records exact desired digest/config and audit trail. Do not type an unverified tag during an incident.

Rollback validation

After rollback:

desired/ready replicas and exact digest match;
error/latency/queue/recovery stabilize;
new writes/events remain readable;
no stale candidate worker can commit (lease generation fences it);
in-flight jobs reconcile;
model/index/harness compatibility is restored;
migration/backfill state is safe;
telemetry is present;
incident evidence is preserved.

Rollback completion is a new observed state, not “command returned success.”

Runnable end-to-end evidence

Host gates:

cd rust/rust-ai-engineering
cargo test -p mosaic-api --all-targets
cargo clippy -p mosaic-api --all-targets -- -D warnings
cargo build --locked --release --package mosaic-api --bin mosaic-service

Real socket smoke:

MOSAIC_BIND_ADDR=127.0.0.1:18080 \
MOSAIC_DRAIN_SECONDS=2 \
./target/debug/mosaic-service

curl --fail http://127.0.0.1:18080/health/startup
curl --fail http://127.0.0.1:18080/health/ready

The verified run accepted one bounded request with 202, then SSE replayed:

run_started
run_failed code=model_provider_not_configured

This proves the lifecycle worker does real state work and reports its current limitation.

Container smoke:

docker build -f containers/Dockerfile -t mosaic-service:lifecycle .
docker run --detach --name mosaic-service-lifecycle-test \
  --read-only --cap-drop ALL --security-opt no-new-privileges \
  -p 127.0.0.1:18081:8080 \
  mosaic-service:lifecycle
curl --fail http://127.0.0.1:18081/health/ready
docker stop --timeout 3 mosaic-service-lifecycle-test
docker logs mosaic-service-lifecycle-test
docker rm mosaic-service-lifecycle-test

--timeout 3 in that smoke is shorter than the binary’s configured 20-second drain, but no jobs were active and it exited immediately. A real drain test sets the container timeout greater than the configured application drain and injects controlled long work.

The Kubernetes YAML parses as three documents (Deployment, Service, PodDisruptionBudget). Local kubectl --dry-run=client may still require API discovery without a configured cluster; schema validation belongs in CI using a pinned validator/Kubernetes version and policy tests.

Failure investigation: new pods never become ready

pause rollout; retain old capacity;
inspect startup/readiness status, events and candidate logs/traces;
confirm exact image/config/secret/model/schema identities;
check bind, worker ownership, migrations, database/object/provider access and model load;
compare resource requests/limits, scheduling, OOM and probe timing;
reproduce the candidate image with production-like config safely;
fix or roll back digest/config without weakening readiness to green.

Increasing probe timeout is correct only when measured valid startup exceeds it.

Failure investigation: rolling shutdown loses requests

distinguish requests never accepted, accepted only in memory, and durably committed;
reconstruct endpoint termination, TERM, readiness, listener and force-kill timing;
inspect keep-alive/SSE/ingress retries and idempotency;
inspect queue sender closure, worker drain report, lease/event history;
compare drain components with pod grace period;
make acceptance durable before 202 and fence recovery;
test delete/scale/rollout/node-loss paths with controlled in-flight work.

A longer sleep cannot repair non-durable acceptance.

Failure investigation: rollback does not recover quality

verify actual pod image digest, config and traffic route;
compare model/tokenizer/preprocessor/harness/prompt/tool/policy/index identities;
inspect database/event/artifact writes made by the candidate;
stop candidate claims and fence stale generations;
route a known-good coherent bundle;
rerun frozen eval/smoke cases and production slices;
forward-fix incompatible data before resuming broad traffic.

Code rollback alone cannot undo an index or prompt rollout.

Failure investigation: liveness restart storm

pause rollout/autoscaling changes and observe restart reason;
inspect liveness latency/timeouts separately from readiness/dependency health;
correlate CPU throttling, memory pressure, queue depth and node conditions;
verify health handler/executor has enough resources to respond;
stop using shared dependency success as liveness;
add startup grace and realistic thresholds;
load-test probe behavior at saturation and backend outage.

Restarts add load and cold starts; they are not generic healing.

Alternatives and trade-offs

Static/musl image

Very small runtime and fewer dynamic packages. Native AI/media/database dependencies, DNS/TLS, allocator/performance and target support can complicate it. Test supported targets.

Minimal Debian runtime

Familiar glibc/native compatibility and package ecosystem. Larger surface than distroless/static and requires base update/scanning.

Mutable tags

Automatic rebuilds may receive fixes but candidate identity drifts invisibly. Resolve updates in reviewed PRs and deploy digests.

Rolling update

Incremental and capacity-efficient; requires mixed-version compatibility and can expose users while promoting.

Blue/green

Fast traffic switch/rollback and environment isolation; doubles capacity and does not solve shared data/schema/model side effects.

Canary

Limits initial exposure and supports evidence comparison; needs representative assignment, enough traffic, clean telemetry and automated containment.

Complete in-flight work on shutdown

Reduces retries for short bounded work. Long AI/media tasks may exceed grace and block rollout. Durable lease recovery remains required.

Immediately release/checkpoint

Speeds termination; checkpoint semantics are complex and external side effects may be uncertain.

Security and performance implications

Security:

pin, update, scan and verify base/application images and build frontend;
build from reviewed source with least-privilege CI and no production secrets;
attach/review SBOM and provenance; sign and enforce admission policy;
run non-root, drop capabilities, disallow escalation, read-only root, default seccomp;
disable unnecessary service-account token and restrict network/egress;
inject scoped secrets at runtime, never image layers;
expose minimal health data;
keep image/debug/core-dump/telemetry/artifact access controlled;
treat model/media/native assets as supply-chain inputs too.

Performance/reliability:

size startup/probe/drain/grace from measured distributions;
pre-pull/cache large images/models without changing identity;
budget rollout surge for CPU/RAM/GPU/provider quotas;
load-test instrumentation and probes enabled;
make accepted work durable and retries idempotent;
preserve queue/capacity headroom during rollout;
canary across modality/input-size/cold-cache slices;
measure cost/quality, not only HTTP success;
test OOM/SIGKILL/node loss, where graceful code does not run.

Exercises

Replace the unconfigured lifecycle worker with the durable SQL job claimant while keeping the same shutdown report. Prove accepted work survives SIGKILL.
Add a controlled 500 ms worker and integration test: enqueue three jobs, signal drain, prove all terminalize before a two-second deadline and new admission is rejected.
Add a long SSE connection and divide connection, worker and telemetry drain deadlines explicitly.
Generate an SBOM/provenance for the image, sign it in an isolated registry, and define admission verification by issuer/repository/workflow/digest.
Add CI policy that rejects the Kubernetes image marker, mutable tags, root user, writable root filesystem, capabilities and missing resource/probe fields.
Design expand/migrate/contract for changing RunEvent with an old/new rolling deployment.
Design a GPU rollout when three replicas occupy all three GPUs and model load takes four minutes.
Create a canary matrix for text, image, audio and video including quality, latency, memory and cost gates.
Rehearse rollback after the candidate wrote a new artifact manifest version.
Chaos-test pod deletion, process crash, OOM and node loss; state which graceful hooks execute.

Solution guidance

The durable worker stops claims on drain, finishes/releases current leases within a sub-deadline and lets generation fencing protect recovery. The controlled-worker test starts the server on an ephemeral port, synchronizes accepted count, triggers a programmatic shutdown token, checks readiness/admission immediately, then joins every child. SSE drain sends a reconnect/last event ID policy before its deadline and leaves time for workers/export.

Image policy parses OCI/Kubernetes references and security contexts in CI; string search alone is weak. SBOM/provenance/signature verification uses pinned tools and a disposable test registry before production policy. Event schema expansion adds optional/versioned fields old consumers ignore, dual-read/write only as required, backfills, then contracts later. GPU rollout needs temporary capacity, blue/green/external shared service, or explicitly accepted unavailability—surge cannot schedule imaginary GPUs. Canary assignment is stable and stratified by modality/size/risk/cold state. Artifact rollback retains readers for both versions or performs a tested forward conversion. SIGTERM exercises graceful logic; SIGKILL/OOM/node loss prove durable recovery instead.

Check your understanding

Why should the same image digest move across environments?
What does digest pinning fail to prove?
Why is the build frontend pinned too?
What must become true before startup health succeeds?
When should readiness fail but liveness remain healthy?
Why can a dependency-backed liveness check cause a restart storm?
What admission race remains when draining begins?
Why is durable acceptance required even with graceful shutdown?
How must application drain relate to pod termination grace?
What does a PodDisruptionBudget not protect?
Why can maxSurge: 1 fail for GPU workloads?
What does a Deployment rollback actually restore?
Why can a prompt/index/model change require compatibility planning?
What proves rollback succeeded?

Containers, Health, Shutdown, Rollout, and Rollback

Learning objectives

Dependencies and mental model

Step 1: define the release unit

Step 2: build a small pinned container

Why copy manifests and crates together?

Runtime contents

Build secrets

Multi-platform images

Step 3: harden runtime defaults

Step 4: health endpoints state facts, not optimism

Startup

Readiness

Liveness

Health response safety

Step 5: make state transitions atomic and test them

Step 6: order graceful shutdown correctly

The deadline hierarchy

Durable versus in-memory work

Side effects during forced drain

Step 7: understand pod termination

Step 8: the Kubernetes rollout template

Image marker

Probes

Resources and scheduling

Step 9: schema and mixed-version compatibility

AI compatibility is schema compatibility

Step 10: staged rollout

Promotion criteria

Step 11: rollback is a designed operation

Rollback validation

Runnable end-to-end evidence

Failure investigation: new pods never become ready

Failure investigation: rolling shutdown loses requests

Failure investigation: rollback does not recover quality

Failure investigation: liveness restart storm

Alternatives and trade-offs

Static/musl image

Minimal Debian runtime

Mutable tags

Rolling update

Blue/green

Canary

Complete in-flight work on shutdown

Immediately release/checkpoint

Security and performance implications

Exercises

Solution guidance

Check your understanding

Completion checklist

Authoritative references