Model Metadata, Version Pins, and Provider Change

Your application requests builder-chat. Yesterday that route resolved to one model release, tokenizer and context limit. Today the provider moves the alias, changes a processor and removes a capability. The HTTP request still succeeds. The type name still says String. Your behavior has changed.

A model name is a routing instruction. It is not sufficient release identity.

This chapter adds versioned release snapshots and a deterministic change audit to mosaic-models. It handles both local artifacts and hosted providers without claiming they offer equal reproducibility. A content-verified local bundle can be pinned by bytes. A provider version is an external assertion. An unversioned alias is only an observation and remains weak even when its text has not changed.

The implementation separates:

logical route: what the product asks for;
requested model: the provider-facing selector;
resolved model: what the runtime reports actually served;
revision evidence: how strongly that release is pinned;
processor, template and runtime metadata;
application-visible capability and context contracts;
observation time: when metadata was acquired, not what the release is.

Learning objectives

By the end you will be able to:

distinguish a product route, provider alias, resolved release and content digest;
rank content hashes, immutable source commits, provider versions and unversioned aliases without pretending they are equivalent;
explain why a Git commit may still not identify every fetched byte;
record processor/tokenizer, prompt template, runtime fingerprint and capability metadata;
compute an order-independent stable identity;
exclude acquisition timestamps from behavior identity;
compare two snapshots for behavioral and breaking changes;
block capability removal, context reduction and weakened pin evidence;
require evaluation for model, provider, processor, template or runtime change;
explain why an unchanged weak alias still requires evaluation;
record actual release metadata on every run;
design canary, shadow, rollback and deprecation workflows;
diagnose provider drift separately from data drift and harness drift;
protect model downloads and metadata as supply-chain inputs;
state the limits of reproducibility for opaque hosted systems.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-models --example release_change_audit

The example compares:

baseline: release-baseline.json
candidate: release-candidate.json
route:     primary-text

The audit finds:

{
  "baseline_pin_strength": "provider_version",
  "candidate_pin_strength": "provider_version",
  "changes": [
    { "type": "resolved_model" },
    { "type": "revision" },
    { "type": "runtime_fingerprint" },
    { "type": "processor" },
    { "type": "prompt_template" },
    { "type": "context_reduced", "before": 8192, "after": 4096 },
    { "type": "capability_removed", "capability": "reranking" }
  ],
  "promotion_gate": "blocked"
}

The full JSON contains before/after values and two 64-hex release identities. “Blocked” does not claim the candidate is universally worse. It means the declared contract shrank, so it cannot silently replace the baseline. A reviewed migration could change the route contract and proceed through new evaluation.

A name is not an identity

Use these terms precisely:

Logical route

An application-owned name such as:

primary-text
fast-router
local-embedding

The route belongs in product configuration. It can select releases by tenant, region, risk, feature flag or experiment. Changing its target is a deployment.

Requested model

The exact selector sent to a provider/runtime:

builder-chat
org/model-name
/models/model.gguf

It might be an alias, tag, path or version. Record it, but do not infer immutability from its spelling.

Resolved model

The concrete release name reported by the runtime/provider, when available:

builder-chat-2026-07-01

This can reveal alias movement. It is still provider metadata, not byte verification.

Revision evidence

The strongest available claim about what executed:

pub enum RevisionEvidence {
    ContentSha256(String),
    GitCommit(String),
    ProviderVersion(String),
    UnversionedAlias,
}

The enum prevents "latest" from being stored in a field called immutable_revision.

Pin strength is an evidence ladder

The companion orders evidence:

unversioned alias
  < provider version
  < immutable Git commit
  < verified content SHA-256

This order is an application policy, not a universal cryptographic theorem. Each step means:

Unversioned alias

You know only the selector used. Identical text at two times does not prove identical behavior. The audit therefore returns rerun_evaluation even when two weak snapshots have no visible changes.

Provider version

The provider exposes a version identifier. This is better change evidence than an alias, but the provider controls its semantics and infrastructure. Read its lifecycle/deprecation contract. A runtime fingerprint can add another change signal but does not turn an opaque service into a content-addressed artifact.

Immutable Git commit

A repository commit strongly pins tracked source metadata. It may not by itself prove the bytes executed:

large files may be stored through an external LFS/object system;
build flags and dependencies can vary;
mutable containers/base images can change;
runtime conversion can create new artifacts;
a remote service may not execute that commit.

Resolve and verify every referenced artifact.

Content-verified digest

A cryptographic digest of the exact activated bytes is the strongest evidence in this ladder. The model-bundle chapter hashes weights, graph, tokenizer, template, license and model card. It still does not guarantee deterministic output: runtime, device, sampling and concurrency remain part of the experiment.

The distinction is the heart of honest engineering:

identity evidence != behavioral evaluation != deterministic execution

You need all applicable layers.

The release snapshot

The checked structure is:

pub struct ModelReleaseSnapshot {
    pub schema_version: u16,
    pub route_id: String,
    pub provider_id: String,
    pub requested_model: String,
    pub resolved_model: String,
    pub revision: RevisionEvidence,
    pub runtime_fingerprint: Option<String>,
    pub processor: ProcessorIdentity,
    pub prompt_template_sha256: String,
    pub maximum_context_tokens: u64,
    pub capabilities: Vec<ModelCapability>,
    pub observed_at: String,
}

It is a deliberately compact teaching contract. Production snapshots may also need:

endpoint region and API version;
input/output modality and media limits;
tokenizer vocabulary/special-token IDs;
generation defaults and supported parameters;
embedding dimensions/normalization/roles;
classifier label map;
safety-policy/filter version;
data residency and retention flags;
runtime/container/device/kernel release;
adapter and quantization identities;
license/use restrictions;
pricing snapshot;
deprecation date;
provider request/response schema revision.

Do not add an unbounded metadata map and call the problem solved. Fields that affect compatibility should be typed, validated and compared.

Step 1: parse strictly, then validate

The JSON structure uses:

#[serde(deny_unknown_fields)]

Unknown fields fail instead of vanishing during deserialization. This catches:

misspelled contract fields;
newer schemas consumed by older code;
unexpected producer output;
attempts to inject an ignored “trust me” flag.

Validation enforces:

schema version exactly 1;
bounded visible-ASCII route/model/provider tokens;
exact lowercase digest/commit lengths;
non-zero context limit;
non-empty unique capabilities;
bounded processor/runtime metadata;
an explicit UTC observation string ending in Z.

The timestamp check is intentionally syntactic and modest. A production format should parse an RFC 3339 type, enforce range/offset policy and use a trusted clock. Do not confuse a well-formed timestamp with trustworthy origin.

Tagged evidence, not magic strings

JSON distinguishes:

{"kind": "content_sha256", "value": "...64 hex..."}
{"kind": "git_commit", "value": "...40 hex..."}
{"kind": "provider_version", "value": "chat-2026-07-01"}
{"kind": "unversioned_alias"}

Each variant gets the right validation and pin strength. A 40-character provider label is not silently treated as a Git commit.

Processors are also explicit:

{"kind": "pinned_revision", "value": "tokenizer-2026-06-20"}
{"kind": "provider_managed"}

provider_managed is honest but weaker. It tells downstream code not to claim a pinned tokenizer.

Step 2: hash behavior identity, not the observation

ModelReleaseSnapshot::identity uses SHA-256 over length-prefixed fields:

len(field name) || field name || len(value) || value

Length prefixes prevent ambiguous concatenations. Capability enums are sorted before hashing, so JSON list order does not change identity.

The identity includes:

route, provider, requested and resolved model;
revision kind and value;
runtime fingerprint;
processor kind and value;
prompt-template digest;
maximum context;
sorted capabilities.

It deliberately excludes observed_at.

If the same strong release is observed on July 28 and July 29, its identity should remain the same. The timestamp remains provenance on the snapshot record. Tests reverse capability order and change the observation time while proving identity remains stable.

This separation is useful beyond models:

release identity: fields that define executable behavior/contract
acquisition record: when, where and by whom those fields were observed

Hash algorithms and field encodings must be versioned. Never change them silently while retaining the same schema_version.

Step 3: compare snapshots on one logical route

The audit refuses to compare different routes. fast-router and primary-text may have different contracts; treating one as a candidate release for the other is an evaluation setup error.

For the same route, it records changes to:

provider;
requested model;
resolved model;
revision evidence;
runtime fingerprint;
processor;
prompt-template digest;
maximum context;
capability additions/removals;
pin strength.

Capability order does not matter. Additions and removals are computed as set differences and emitted in stable enum order.

The change list is data. CI, a release reviewer and incident tooling can inspect exactly what changed instead of parsing a prose diff.

Step 4: choose a promotion gate

The teaching policy has three outcomes:

pub enum PromotionGate {
    ExactRelease,
    RerunEvaluation,
    Blocked,
}

`exact_release`

Returned only when:

no behavior/contract field changed; and
revision evidence is stronger than an unversioned alias.

This means “same declared release,” not “skip every operational check forever.” Data, traffic and dependencies outside this snapshot can still drift.

`rerun_evaluation`

Returned when:

provider/requested/resolved model changes;
revision or runtime fingerprint changes;
processor or prompt template changes;
context increases;
a capability is added;
or no visible field changes but the release uses an unversioned alias.

These may be good changes. They need the relevant held-out, safety, latency and cost gates before promotion.

`blocked`

Returned when:

maximum context decreases;
a declared capability disappears;
pin evidence weakens.

Those are contract regressions in this route. A human cannot approve the same release blindly. The team can intentionally define a new contract, update dependents, rerun evaluation and migrate.

Blocking a context decrease matters even if ordinary prompts are short. A previously accepted request can now fail or truncate evidence. Blocking capability removal prevents a reranker endpoint from quietly becoming generation-only.

Why context increase still needs evaluation

A larger advertised window is not automatically better:

useful-context quality may degrade at length;
attention/cache cost grows;
truncation behavior changes;
admission concurrency changes;
new input may expose prompt-injection surfaces;
provider billing changes.

It is non-breaking at the shape level but behavior-affecting.

Provider observation workflow

At configuration time

Record the intended route contract:

required capability
minimum context
allowed provider/region
minimum pin strength
data-handling requirements
approved release identity

Reject startup when required metadata is absent or incompatible. Do not wait for the first customer request.

On every response

Capture safe provider-returned metadata:

actual model/release name;
runtime/system fingerprint when offered;
API version;
usage units;
stop reason;
request ID for provider support;
region/route when policy permits;
locally selected adapter/template identities.

Attach the computed release identity to the internal trace. Do not rely only on dashboard configuration; actual routing may fall back or canary.

Provider request IDs may be sensitive or high-cardinality. Keep them out of metric labels and store them in protected traces with retention bounds.

On mismatch

Fail or quarantine according to risk:

unexpected model on low-risk summarization
  → mark run, stop promotion, alert, possibly serve with warning/fallback

unexpected model on irreversible tool action
  → fail closed before authorization/execution

Do not silently rewrite the expected snapshot to match production. That erases the evidence needed to investigate.

Local artifact workflow

For locally controlled inference:

Resolve repository/tag to an immutable commit.
Download into a staging directory with byte/file limits.
Verify expected content digests and reject symlinks/traversal.
Validate graph/config/tokenizer/template compatibility.
Convert/quantize reproducibly when possible.
create a new manifest for converted bytes; do not reuse the source digest.
Run smoke, quality, safety and performance gates.
Atomically activate the approved release.
Keep the previous verified bundle for rollback.

Never execute arbitrary remote model code merely because a registry page suggests it. Review and pin code, isolate conversion, minimize permissions and treat model formats/parsers as attack surfaces.

Hosted provider workflow

Hosted services expose less:

Prefer dated/explicit versions over moving aliases.
Snapshot all documented metadata the API returns.
Record which fields the provider promises are stable.
Detect alias/resolved-model/fingerprint changes.
Run scheduled canaries even if metadata is unchanged.
maintain a tested fallback with compatible data policy and task contract.
Track deprecation notices and migration deadlines.
Keep release-eval evidence for both current and candidate versions.

An opaque provider can change serving kernels, safety layers or routing without exposing byte identity. State this limitation. Behavioral canaries and production monitors complement metadata; they do not manufacture reproducibility.

Change is larger than the weights

A proper release identity spans:

model artifact / provider release
  + tokenizer / media processor
  + chat or task template
  + adapter / quantization
  + inference runtime and relevant defaults
  + capability schema
  + harness prompt/tools/retrieval/policy
  + evaluation suite and grader releases

This chapter snapshots the model-facing subset. Later harness chapters create a complete system release. If model identity is pinned but the prompt template floats, behavior still floats. If the prompt is pinned but retrieval index silently re-embeds with another model, behavior still floats.

Do not use “model version” when you mean “product behavior version.”

Task-specific metadata that must travel with identity

A generic capability name is not enough. Each contract from the previous chapter has metadata that changes the meaning of otherwise valid numbers.

Generation

Record:

tokenizer + special-token map
chat/task template
generation defaults
supported logit processors
EOS/stop behavior
maximum input/output/total tokens
structured-output/tool schema support

A change to the EOS ID can turn a healthy model into one that runs to every length limit. A new template can alter role precedence without changing weights.

Embeddings

Record:

dimension + component dtype
pooling/projection
normalization
distance metric
query/document prompts or routes
maximum input and truncation

An index must reject a release mismatch even when dimensions happen to agree. Keep an index-version pointer rather than overwriting old vectors in place.

Reranking

Record:

pair serialization/order
maximum pair length + truncation
raw output semantics
optional activation/calibration map
candidate limit and stable tie policy

Changing (query, candidate) to (candidate, query) can change output while preserving every tensor shape.

Classification

Record:

ordered label-ID map
single-label vs multi-label objective
softmax/sigmoid/none activation
per-label thresholds
calibration-map identity

Thresholds are product policy and may be deployed separately from weights. They still belong in the complete decision-system release and evaluation comparison.

Release snapshot, run record, and approval are different artifacts

Avoid one mutable “model metadata” row.

ModelReleaseSnapshot says what release/contract was observed. A RunRecord says what actually happened:

run/task/tenant-safe IDs
release identity
route and actual resolved release
prompt/tool/retrieval/policy release identities
sampling and seed evidence
input/output usage
stop/terminal reason
latency and safe trace references

An ApprovalRecord says what evidence authorized deployment:

candidate release identity
baseline identity
evaluation suite + dataset + grader digests
quality/safety/performance report digests
known limitations and accepted exceptions
reviewer and timestamp

All three should be append-only or content addressed according to risk. Deployment state can point to an approval; it should not rewrite the approval when a route changes.

Comparisons are valid only when their evaluation identity is also comparable. If a candidate uses a new grader or cases, report both:

baseline and candidate on the same frozen suite for a paired comparison;
both releases on the new suite for forward-looking coverage.

Running the baseline only on yesterday’s suite and the candidate only on today’s suite confounds release change with evaluation change.

Rollout and rollback

A safe provider/model update:

candidate snapshot acquired and validated
  → change audit
  → offline eval on frozen cases
  → shadow traffic with privacy controls
  → small canary
  → compare quality/safety/latency/cost slices
  → staged increase
  → full promotion
  → retain rollback and monitor

Use stable assignment for A/B traffic so one user does not oscillate releases. Carry release identity through queue, cache and persistence. An old queued job should either execute its recorded release or be explicitly migrated—not accidentally use today’s route.

Cache keys must include behavior identity. Otherwise an answer, embedding or rerank result from one release can be served as if produced by another.

Rollback must restore:

model/provider route;
processor/template;
compatible index/schema;
calibration thresholds;
safety policy;
capacity configuration.

Rolling back only the weight filename is incomplete.

Drift taxonomy

When quality moves, locate the changed layer:

Drift type	Example evidence
Release drift	resolved model, revision, processor or fingerprint changed
Harness drift	prompt/tool/policy digest changed
Data drift	input distribution/slice proportions changed
Knowledge drift	world facts changed while model remained fixed
Retrieval drift	corpus/index/query encoder changed
Runtime drift	kernels, dtype, quantization or device changed
Grader drift	labels/rubric/judge release changed

Several can occur simultaneously. Record identities so the investigation can stratify results.

Security and privacy

Release metadata is part of the supply chain:

authenticate registry/provider endpoints;
verify digests after download;
allowlist schemes/hosts and block local-network SSRF;
bound files, bytes and decompression;
reject path traversal/symlinks;
scan and sandbox parsers/converters;
do not expose provider credentials in snapshots;
sign provenance where organizational policy requires it;
restrict who can change production routes;
audit approvals and rollbacks.

Public model names can still reveal product strategy; internal endpoint names and fingerprints can reveal infrastructure. Classify and redact appropriately.

Failure investigation

“Nothing changed, but scores regressed”

Check whether the snapshot used an unversioned alias, whether actual response metadata was captured, and whether prompt/retrieval/grader/data changed. “Same alias” is not evidence of same release.

Identity changes every observation

Verify that timestamps, request IDs, latency and transient endpoint IDs are excluded from release identity. Keep them in the acquisition/run record.

Identity stays the same after template change

The template digest is missing from the identity or the deployed renderer is not the file being hashed. Hash the exact activated bytes.

Candidate fails only long prompts

Inspect context reduction, tokenizer changes, reserved output, template overhead and truncation. Advertised context is not the same as useful or admitted context.

Rollback restores model but not quality

Check processor, prompt, retrieval index, calibration map, runtime and safety configuration. Restore the complete approved system release.

Exercises

Exercise 1: classify evidence

Rank these selectors:

main
v2
4f91... (40-hex commit)
sha256:ab31... (verified exact activated bytes)
provider model dated 2026-07-01

Solution guidance

main is a moving branch. v2 is a tag/name unless the registry guarantees immutability. A 40-hex commit pins repository state but not necessarily external artifacts/build output. A dated provider release is a provider assertion. A digest verified over exact activated bytes gives the strongest content evidence. The provider release and Git commit are not universally orderable without their contracts; the companion chooses an explicit policy and does not call either content verified.

Exercise 2: extend the snapshot

Add embedding dimension, normalization and query/document role. Emit distinct changes and block a dimension change on an existing retrieval route. Test capability order independence.

Exercise 3: sign an approval

Define an ApprovedRelease that binds route, release identity, eval-report digest, approver identity, decision timestamp and signature. Keep mutable deployment status outside the signed payload.

Exercise 4: queue identity

Add release identity to a durable job. Test that a worker with a different active release rejects or explicitly migrates it. Ensure retry does not silently switch versions.

Exercise 5: weak provider canary

Design a 20-case canary that runs hourly against an unversioned alias. Include deterministic structure, refusal/tool policy, one shifted fact and latency. Define the alert and failover rules without logging sensitive prompts.

Exercise 6: complete rollback

Write a table listing model, tokenizer, template, retrieval index, policy, calibration and capacity versions for baseline/candidate. Simulate rollback and prove every pointer returns to the baseline.

Check your understanding

Why is requested model different from resolved model?
Why is a provider version not equivalent to a verified content digest?
Why must observation time be excluded from release identity?
Why does an unchanged unversioned alias still need evaluation?
Which changes does the teaching policy block immediately?
Why can increasing context affect quality and cost?
What model metadata belongs on every run trace?
Why must queued work and cache entries carry release identity?
What else must rollback restore besides weights?

Completion checklist

Sources and further study

Hugging Face Hub, Downloading files: revision selectors can be branches, tags or commit hashes; choose the immutable evidence your workflow requires.
Hugging Face Transformers, Loading models: examples of pinning custom model code with a commit revision.
MLflow, Model Signatures and Input Examples: explicit input, output and parameter schemas as model interface contracts.
NIST, AI Risk Management Framework 1.0: lifecycle-wide measurement, monitoring and management of AI risk.
NIST, AI RMF Playbook: post-deployment monitoring, incident response, recovery and change-management practices.
The previous model-artifact chapter: content verification, component completeness, provenance and activation boundaries.
The previous sampling chapter: why pinned artifacts do not make all executions bit-reproducible.

What this chapter does not claim

SHA-256 proves equality to expected bytes, not that those bytes are safe, high quality or licensed for the use. A provider version/fingerprint does not reveal opaque serving internals. Snapshot identity does not detect changes in user traffic, world knowledge, retrieval data or an untracked harness. ExactRelease means the declared strong model-facing contract is unchanged; it is not a blanket approval to bypass operational monitoring.

The next chapter compares remote APIs, embedded runtimes and separate inference services so release identity survives different process and trust boundaries.