Day 24 — Follow Tokens to Logits

Today Model Lab follows one tiny request through a decoder-only path.

Run the mechanics

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example causal_mechanics
cargo run -q -p mosaic-ml --example sampling_replay

The numbers are fixed didactic weights, not a trained model. That is the point: you can inspect every matrix and prove causality.

Name every axis

token IDs                 [sequence]
embedding rows            [sequence, model_width]
attention scores          [sequence, sequence]
hidden states             [sequence, model_width]
vocabulary logits         [sequence, vocabulary]

Use constructors that validate lengths, shapes, finite values, and vocabulary bounds. A flat Vec<f32> without shape metadata is easy to multiply incorrectly and still produce plausible numbers.

Prove the causal invariant

Changing a future token must not change earlier hidden states. Write a test that evaluates two sequences sharing a prefix and compares outputs before the changed position. If it fails, inspect the mask before tuning prompts.

Separate logits from sampling

The model produces logits. A sampling policy applies temperature, top-k/top-p, seed, and stop rules to choose a token. Record both model release and sampling configuration. Same prompt plus same seed is not a universal reproducibility promise across changing providers or kernels.

Break it deliberately

• out-of-vocabulary token ID;
• matrix dimension mismatch;
• all-masked attention row;
• unstable softmax overflow;
• NaN logit;
• future-token leakage.

Each becomes a typed numeric or contract error rather than mysterious nonsense text.

Completion proof

Explain one generation step from UTF-8 input to chosen next-token ID and identify which stages belong to the model runtime versus the application harness.

Deep reference: Tokens, embeddings, tensors, and transformers.

Next: Day 25 — Local Inference →.