Sampling, Seeds, and Reproducibility Boundaries

A model forward pass returns logits. The model has not yet “chosen a word.” The generation policy must transform one vocabulary-sized score vector into the next token ID:

logits [V]
  → ordered logit processors
  → temperature
  → truncation / constraints
  → normalized candidate distribution
  → greedy choice or PRNG draw
  → token ID

Changing this policy can change output even when model weights and input are identical. Conversely, holding a seed constant does not guarantee identical output when model logits change by a tiny amount, a provider changes model revisions, a GPU kernel reduces values in a different order, or a concurrent service consumes random numbers in a different schedule.

This chapter adds a versioned sampler to mosaic-ml. It implements strict greedy selection, temperature, top-k and top-p/nucleus truncation, stable probabilities, a specified SplitMix64 random stream, exact draw indices and deterministic tie-breaking. Then it draws the crucial boundary:

A seed can replay one pseudorandom stream. It cannot freeze the entire inference system.

Learning objectives

By the end you will be able to:

distinguish logits, probabilities, token selection and sequence decoding;
explain greedy, temperature, top-k and top-p selection mathematically;
describe why sampling transforms are ordered and why changing order changes behavior;
implement stable softmax and renormalization after truncation;
handle equal logits with an explicit token-ID tie break;
use a local per-run PRNG instead of one process-global random stream;
record algorithm version, seed, draw index and generation settings for replay;
distinguish PRNG replay, bitwise numerical reproducibility, output reproducibility and semantic repeatability;
enumerate sources of nondeterminism beyond the sampler;
explain why greedy decoding can still differ across backends;
design deterministic tests without overclaiming production determinism;
evaluate generation settings on task distributions rather than choosing “creative” numbers by folklore;
protect seeds from becoming mistaken for security randomness;
define what a provider’s seed feature can and cannot promise.

Dependencies and completion artifact

This chapter depends on:

the prior logit/softmax/transformer mechanics;
strict configuration types and finite-number checks;
model, tokenizer, template and runtime release identity;
traces and replay as product concepts.

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example sampling_replay

The checked-in example uses logits [0,1,2,3], seed 42, temperature 0.8, top-k 3, and top-p 0.9. It currently emits:

{
  "algorithm_version": "mosaic-splitmix64-top-k-top-p-v1",
  "seed": 42,
  "tokens": [3, 3, 3, 3, 3, 2, 3, 2],
  "scope": "replayable for this sampler contract; not a universal model reproducibility claim"
}

The full output includes eight uniform draws. The example repeats one fixed logit vector to isolate the sampler. Real autoregressive generation feeds each chosen token back into the model, so one different token changes subsequent logits and can make the remaining sequence diverge.

Mental model: four layers of repeatability

Use precise language:

PRNG-stream replay

Given the same PRNG algorithm/version, seed, state-transition rules and number/order of draws, the generator emits the same integer/random stream. The companion proves this inside one specified contract.

Bitwise numerical reproducibility

Given identical input tensors and release environment, every inference operation emits identical bits. This is stronger. Floating-point operations are not associative; different reduction order, fused operations, kernels, devices, batch shapes or library versions can alter low bits.

Token/output reproducibility

The complete generation emits the same token IDs/bytes. This can hold even without bitwise-identical logits if numerical differences do not cross a decision boundary. It can fail dramatically if two near-tied candidates swap order.

Semantic repeatability

The output may differ in wording but retain the required facts, schema and task quality. For many AI products this is the operationally meaningful level. It is measured by evals, not byte equality.

A production statement should say which level, environment and tolerance it means:

Not:  “seeded requests are deterministic”

Better:
“For release R on runtime/device class E, the replay suite reproduces exact token IDs for
these frozen cases. Cross-release/platform output is evaluated semantically and is not
promised bit-for-bit.”

PyTorch’s reproducibility guidance makes the same important limitation: complete reproducibility is not guaranteed across releases, commits, platforms, or CPU versus GPU even with identical seeds.

Step 1: greedy selection

Greedy decoding chooses the largest logit:

next_id = argmax(logits)

It does not require softmax because softmax preserves ordering. The companion returns no selected_probability for greedy choice; computing a distribution would add work and could imply that greedy selection sampled from it.

Equal maxima need policy. The implementation chooses the lowest token ID:

let token_id = logits
    .iter()
    .enumerate()
    .max_by(|(left_id, left), (right_id, right)| {
        left.total_cmp(right)
            .then_with(|| right_id.cmp(left_id))
    })
    .map(|(token_id, _)| token_id)
    .expect("validated logits are non-empty");

The reversed ID comparison inside max_by makes the lower ID win. This is tested. Without an explicit tie rule, iteration order, parallel reductions or an unstable sort can select a different token.

Greedy is useful for:

narrow extraction/classification-like generation;
deterministic-enough regression fixtures on one controlled backend;
structured output when combined with schema-constrained decoding;
latency-sensitive tasks where multiple candidates bring no measured benefit.

It is not automatically accurate. Repeated local maxima can produce bland, repetitive or brittle sequences. Beam search and other maximization methods have their own task-dependent trade-offs.

Why greedy is not a universal determinism guarantee

Suppose logits for IDs 7 and 19 are:

backend A: 4.200001, 4.200000
backend B: 4.200000, 4.200001

Both are numerically close, but argmax differs. The selected token changes the next input, so later logits need not remain close. Greedy removes sampler randomness; it does not remove model/runtime numerical variability.

Step 2: temperature

For positive temperature τ:

scaled_logit_i = logit_i / τ
probability_i = softmax(scaled_logits)_i

τ = 1 leaves relative logits unchanged before softmax.
0 < τ < 1 increases differences and sharpens the distribution.
τ > 1 reduces differences and flattens the distribution.
τ = 0 is undefined division, not a portable spelling for greedy.

The companion uses separate variants:

pub enum SelectionStrategy {
    Greedy,
    Sample(SamplingConfig),
}

SamplingConfig::try_new requires finite temperature greater than zero. This prevents a common configuration ambiguity where one runtime treats zero as greedy, another clamps it, and another produces invalid arithmetic.

Temperature does not inject randomness by itself. It reshapes the distribution; a random draw selects from it.

Numerical stability

The sampler converts finite f32 logits to f64, divides by temperature, subtracts the maximum before exponentiating, sums and normalizes. It rejects a non-finite or zero denominator.

Using f64 here makes the small reference sampler less fragile, not universally identical across math libraries/platforms. A production accelerator sampler may use other precisions and fused kernels. Bind that behavior to runtime release/evals.

Step 3: top-k truncation

Top-k keeps the k highest scaled-logit candidates and removes the rest:

sorted candidates → first k → softmax/renormalize → sample

The companion sorts descending by scaled logit and ascending by token ID for ties. It rejects:

k = 0;
k > vocabulary size.

Some libraries clamp oversized k; strict rejection catches an incorrect model/vocabulary config instead of silently changing policy.

k = 1 is effectively greedy with an unnecessary random draw in many implementations. The companion still treats it as sampling and records a draw because the caller selected the sampling contract. Its test proves the highest logit always wins.

Fixed k ignores the distribution’s changing shape. At one step the top three candidates may hold almost all probability; at another, dozens may be plausible. This motivates nucleus sampling.

Step 4: top-p / nucleus truncation

Nucleus sampling:

sorts candidates by descending probability;
keeps the smallest prefix whose cumulative probability is at least p;
renormalizes that prefix;
samples from it.

p must be in (0,1]. At p=1, all otherwise permitted candidates remain. At lower p, candidate count adapts to distribution concentration.

Holtzman et al. introduced nucleus sampling in response to observed degeneration in open-ended neural text generation. That research does not mean one top-p value is best for every task/model. Extraction, code editing, image-caption grounding and fiction have different objectives and failure costs.

The companion applies:

temperature → descending stable tie order → top-k → softmax → top-p → renormalize → draw

This is part of mosaic-splitmix64-top-k-top-p-v1. Another library may apply filters or penalties in a different order. The same named parameter values then do not guarantee the same distribution.

For logits [4,3,0] with p=0.60, the highest candidate alone already exceeds 0.60, so the nucleus has one member and renormalizes to probability one. The unit test checks that exact boundary.

Step 5: sample with a specified PRNG

After truncation/normalization, draw u uniformly from [0,1) and select the first candidate whose cumulative probability exceeds u:

u = 0.72

candidate A cumulative 0.50 → continue
candidate B cumulative 0.80 → select B

The implementation retains the last candidate as a floating-rounding fallback. Probabilities are validated to sum to one within a tight reference tolerance.

The companion uses SplitMix64:

state += 0x9e3779b97f4a7c15
z = state
z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9
z = (z ^ (z >> 27)) * 0x94d049bb133111eb
output = z ^ (z >> 31)

All integer operations that can overflow use wrapping arithmetic by specification. The top 53 bits become an f64 in [0,1). The algorithm identifier is emitted with every decision/report.

SplitMix64 here is:

compact;
deterministic under the stated integer algorithm;
adequate for a teaching sampler/replay fixture;
not cryptographically secure.

Never use it for API keys, session tokens, nonces, password salts, lottery/security decisions or unpredictable IDs.

Seed and state

A seed initializes state. Re-seeding before every token repeats the first draw and biases behavior. A generation run should normally initialize once, then consume one sampler draw per sampling step under the exact algorithm.

SampleDecision records:

pub struct SampleDecision {
    pub token_id: usize,
    pub candidate_count: usize,
    pub selected_probability: Option<f64>,
    pub random_draw: Option<f64>,
    pub draw_index: Option<u64>,
    pub algorithm_version: &'static str,
}

The draw index catches hidden consumption. If a new logit processor performs a random tie break or a speculative decoder consumes/rejects candidates, subsequent draws can shift even with the same seed.

Exact probability/draw traces may reveal model behavior or sensitive token choices. Keep detailed replay traces under the same access/retention policy as prompts and outputs.

Step 6: isolate random streams per run

A process-global RNG shared by concurrent requests is scheduling-dependent:

run A consumes draw 0
run B consumes draw 1

after a timing change:
run B consumes draw 0
run A consumes draw 1

Locks prevent data races but do not create run-level replay. Give every run its own sampler state. Store the seed (when permitted), algorithm version and draw count in the run trace.

For parent/child operations, derive labeled substreams deliberately:

run seed
  ├─ "draft-model"
  ├─ "verifier"
  ├─ "image-variation-0"
  └─ "repair-attempt-1"

Do not use Rust’s default hash as a stable derivation contract. Specify a KDF/hash, domain separation, exact byte encoding and version. If seeds must be unpredictable or user isolation matters, derive them with an approved cryptographic primitive and secret entropy. If reproducible public experiments matter, fixed public seeds may be appropriate.

Retry semantics must be explicit:

replay retry: same seed/state and same complete input aims to reproduce the failed attempt;
new candidate retry: a distinct derived seed explores another trajectory;
resume: continue from persisted state/draw index only if model/runtime state is also resumable.

Calling all three “retry with seed 42” is insufficient.

Step 7: define the reproducibility envelope

To investigate exact output, record or bind:

Model and input

model bundle/release identity, adapters and quantization;
tokenizer/processor and template identity;
exact rendered input token IDs or protected replay artifact;
retrieved evidence and ordering;
tools/schema/policy/prompt release;
media decoder/normalization outputs.

Runtime

runtime/library/driver versions and build features;
CPU/GPU/accelerator model and relevant device topology;
dtype, quantization and accumulation behavior;
kernels/attention implementation and deterministic settings;
batch shape, padding, sequence lengths and device placement;
compiler flags and target where relevant.

Generation

sampler algorithm/version;
every logit processor and exact order;
temperature, top-k, top-p, penalties, bias, constraints;
seed and PRNG state/draw indices;
stop tokens/sequences, max tokens and EOS behavior;
speculative decoding/beam/search configuration.

Product environment

tool results and external state versions;
clock/random IDs if included in context;
retrieval index snapshot;
cancellation/deadline and race outcomes;
provider returned revision/fingerprint/usage fields;
code/config/container release identity.

If one item is unavailable, state the limit. Honest “semantic replay only” is more useful than a false deterministic claim.

Floating point and backend nondeterminism

Floating-point addition is not associative:

(a + b) + c  may differ from  a + (b + c)

Parallel reductions can reorder operations. Fused multiply-add can round once rather than twice. Different kernels may tile/accumulate differently. NVIDIA’s CUDA guidance explicitly warns that parallel and sequential or CPU/GPU results may differ and recommends tolerance-based comparison where bitwise identity is not meaningful.

Other sources include:

atomic operations whose execution order varies;
auto-tuned kernel selection;
nondeterministic algorithms;
uninitialized memory bugs;
concurrency and dynamic batching;
mixed precision and device-specific math;
compiler/runtime upgrades;
distributed collective order;
provider-side model or infrastructure changes.

Some frameworks expose deterministic-algorithm modes and can error when an operation lacks a deterministic implementation. These modes may reduce performance and still have a scoped platform/release guarantee. Test the exact deployment path.

Beyond top-k and top-p

Production libraries may implement:

minimum probability relative to the maximum;
typical/epsilon/eta truncation;
repetition, frequency and presence penalties;
no-repeat n-grams;
token allow/deny lists and logit bias;
bad-word filters;
forced BOS/EOS or decoder prefixes;
contrastive search or beam methods;
grammar/JSON-schema constrained decoding;
speculative decoding.

These transforms are not commutative. For example, applying repetition penalty before top-k can change which tokens enter the candidate set. A constrained decoder may mask most of the vocabulary after parsing partial JSON state. Therefore release identity must include ordered processors and their versions, not just a flat parameter map.

For tool calls and structured output, constrained decoding can prevent syntactically invalid tokens, but it does not prove semantic validity, authorization, correct tool arguments or factual accuracy. The harness still parses, validates, authorizes and evaluates.

Reproducibility testing strategy

Use multiple gates:

Sampler unit tests

The companion has seven:

greedy ties choose lowest ID without consuming RNG;
same seed/config replays the same decision sequence;
top-k one always chooses the maximum;
top-p can reduce to one candidate and renormalize;
multiple seeds explore both equal candidates;
invalid temperature/k/p fail construction;
empty/NaN logits and oversized top-k fail without consuming a draw.

Numerical reference tests

For tiny frozen tensors, compare runtime logits to a reference within justified absolute/relative tolerance. Also compare top candidate order/margins. Exact equality may be appropriate only within a verified deterministic envelope.

Token replay tests

On one pinned deployment environment, replay exact prompts/seeds and compare token IDs. Capture the first divergence with logits or top candidate summaries, not only the final text diff.

Semantic regression evals

Run multiple seeds/cases and compare quality distributions, schema validity, safety, latency and cost. A single seed can accidentally flatter or punish a release. Paired cases/seeds reduce variance when comparing two harness/model versions.

Metamorphic tests

Check invariants:

top-k one equals argmax;
greedy consumes no PRNG state;
top-p candidate count is at least one;
probabilities are finite/nonnegative and normalize;
disallowed tokens are never selected;
same per-run trace replays independently of request scheduling.

Failure investigations

Failure 1: same seed, different output after deploy

Do not begin by blaming “model randomness.” Compare complete release identities and first differing stage:

rendered bytes/token IDs;
first-step logits or ordered top candidates;
processors and candidate set;
PRNG algorithm/draw index/value;
selected token;
next-step divergence.

If logits differ before sampling, the seed is irrelevant to the root cause. Inspect model/runtime, input composition, batch/device and provider version.

Failure 2: same process is stable; concurrent test is not

Likely a shared RNG, mutable generation config, unordered candidate collection, dynamic batching effect or data race in cache/state. Give each run immutable config and its own sampler. Sort candidates with an explicit tie rule. Trace run-local draw indices and test with forced interleavings.

Failure 3: output becomes unexpectedly repetitive

Possible causes:

temperature/top-p lowered;
top-k accidentally one;
penalties removed or applied after truncation;
stop/EOS mishandled;
wrong tokenizer/template;
model changed;
greedy used for an open-ended task.

Inspect the ordered candidate distribution at a few protected failing steps, compare configuration release and run an eval across cases/seeds. Do not tune one anecdotal prompt.

Failure 4: nucleus emits a token thought to be outside top-p

Clarify semantics: top-p retains the smallest prefix whose cumulative mass reaches or exceeds p. The last included token can push the sum above p. Then probabilities are renormalized. Check sort order, whether top-k/penalties ran first, floating tolerance and tie handling.

Failure 5: a retry is identical when diversity was intended

The code reinitialized the same seed/state for each attempt. Derive a labeled attempt seed or persist and advance a deliberate parent stream. Record lineage so different candidates remain replayable.

Failure 6: provider seed sometimes fails exact replay

Read the provider’s exact contract. A seed may be best-effort and scoped by model/backend fingerprint. Provider-side batching, infrastructure or model updates may remain outside your control. Store returned version/fingerprint fields, monitor replay/eval drift, and promise only the observed level.

Security, privacy and operational implications

A deterministic PRNG is not a security RNG.
User-controlled seeds can be useful for replay but can also probe behavior; rate/authorization still apply.
Do not use output stochasticity as a safety control. Policy must hold across seeds.
Store seed/settings with protected run identity; output and token probabilities may be sensitive.
Bound vocabulary length/candidate allocation and reject non-finite logits.
Avoid logging full logits: they are large, costly and may leak model/data information.
Sampling normally costs far less than model forward pass but sorting all V candidates is nontrivial. Production kernels use partial selection/fusion; benchmark exact path.
Greedy/top-k can avoid full probability materialization depending on needs.
Deterministic modes can reduce throughput; quantify the debugging/regression value and production SLO trade-off.
Continuous batching and speculative decoding improve throughput but complicate exact replay. Treat scheduler/speculation algorithms as release inputs.

Exercises

Exercise 1: trace processor stages

Add a bounded SamplingTrace containing the top five (token_id, logit/probability) entries after each processor, candidate count and selected decision. Redact/disable it by default. Prove trace generation never changes the selection or PRNG stream.

Exercise 2: repetition penalty

Implement one precisely documented repetition-penalty rule using previous token IDs. Decide whether positive and negative logits transform differently, place it in the processor order, and add cases for duplicates, empty history and special tokens. Version the algorithm.

Exercise 3: run-local substream derivation

Define canonical bytes for (root_seed, domain_label, attempt_index) and derive seeds with a cryptographic hash/KDF. Add golden vectors and prove reordering concurrent tasks does not alter their streams. Explain whether the root seed is public or secret in your threat model.

Exercise 4: first-divergence comparator

Given two generation traces, report the first difference among input IDs, logits checksum/top candidates, candidate set, draw index/value, token ID and stop state. Never report sensitive raw values without explicit debug authorization.

Exercise 5: paired sampling eval

Compare two generation configurations over at least 50 frozen cases and five fixed seeds per case. Report paired quality, schema validity, repetition, output length, latency and cost differences with uncertainty—not just the best example.

Exercise 6: constrained JSON decoding

Build a tiny deterministic automaton for {"ok":true|false} that masks invalid next tokens. Show that every emitted sequence is syntactically valid, then add semantic validation proving syntax alone cannot decide whether "ok" is truthful.

Solution guidance

Generate trace views from already computed candidates; never call RNG or re-sort with a different comparator. Cap entry count and protect detailed traces behind policy.
Specify equations before code, apply before truncation if that is the chosen contract, and keep a stable set/history representation. Test ordered processor composition.
Use length-prefixed/domain-separated canonical encoding and an approved hash/KDF. Golden vectors lock the derivation version; per-task labels isolate scheduling.
Compare release metadata first, then steps by sequence number. Distinguish “missing evidence” from “equal.” A logits digest detects change but cannot diagnose which candidate moved, so retain bounded protected top summaries when allowed.
Pair by case and seed; aggregate deltas and failure categories. Keep seed set independent of tuning to avoid overfitting the evaluation.
Maintain parser state and derive an allowed-token mask before sampling. Still parse the completed object with a strict schema and authorize/verify its meaning.

Check your understanding

Does temperature itself make output random?
Why is temperature zero rejected instead of treated as greedy?
What exactly does top-p retain?
Why must probabilities be renormalized after truncation?
What does a seed determine?
Why can greedy decoding differ between CPU and GPU?
How can concurrency break replay even with a locked global RNG?
Why does processor order belong to release identity?
What is the difference between token replay and semantic repeatability?
Is SplitMix64 appropriate for generating password-reset tokens?

Answers:

No. It rescales logits; a selection strategy/draw introduces stochastic choice.
Division by zero is undefined and runtimes differ; a typed greedy mode is unambiguous.
The smallest descending-probability prefix whose cumulative mass reaches or exceeds p.
Sampling requires a distribution summing to one over the retained candidate set.
Initial PRNG state under a particular algorithm; not model/runtime/input or draw schedule.
Floating reduction/kernel/dtype differences can reorder near-tied logits.
Request scheduling changes which run consumes each draw.
Non-commuting transforms can create different candidate distributions.
Token replay requires the same IDs; semantic repeatability permits different wording that meets the same evaluated behavior.
No. Use an approved cryptographically secure generator.

Completion checklist

References

The selection policy is explicit. Next we split runtime work into prompt prefill and token-by-token decode, model the key/value cache, and turn context length, batch, concurrency and output length into capacity and latency budgets.