Structured Outputs and Constrained Decoding

Structured output is not “ask for JSON.” It is a protocol between a probabilistic producer and deterministic software.

A provider can constrain generation so every emitted token remains compatible with a JSON Schema or grammar. That removes a large class of formatting failures. It does not prove that the selected enum is appropriate, a confidence value is calibrated, a citation exists, or a claimed root cause follows from evidence.

The reliable boundary is:

behavior contract
  + exact schema identity
  + evidence-manifest identity
  + provider decoder capability
              │
              ▼
      constrained generation
              │
              ├── typed refusal
              ├── incomplete / filtered
              └── candidate JSON
                       │
        bounded layered validator
        envelope → syntax → structure → semantics → evidence
                       │
              accepted output or typed violations
                       │
              reproducible validation receipt

This chapter replaces the old combined output/tool overview with one focused implementation: mosaic-harness::structured_output and the runnable layered_structured_output example. Chapter 10 goes deeper into semantic and evidence validation. Chapters 11 and 12 build typed tool contracts and untrusted observations on top of this output boundary.

Learning objectives

By the end you will be able to:

distinguish JSON mode, schema validation, and token-level constrained decoding;
make one schema identity flow from contract through decoder and validation receipt;
design closed, bounded, versioned JSON objects;
keep the provider envelope separate from the model-produced payload;
represent refusal, truncation, filtering, and success as non-confusable states;
reject oversized output before parsing;
separate syntax, structure, semantic, and evidence violations;
use strict Serde types without mistaking deserialization for domain validation;
bind citations to an exact evidence manifest and locator set;
detect contract, schema, evidence, generation, and receipt drift;
write mutation tests for every output boundary;
decide when constrained output is a poor fit.

Prerequisites and dependency order

Complete the behavior, run-envelope, prompt-release, provider-request, context-manifest, retrieval, and memory chapters first. They supply the identities this contract must not rediscover:

the behavior digest says what counts as a successful answer;
the prompt release says what instructions and examples were sent;
the provider request says which model route and capabilities were selected;
the context manifest says which evidence was visible;
the evidence manifest gives source-restorable citation targets;
the run envelope supplies output-byte and call budgets;
the trace captures the provider envelope and validation result safely.

Generate schemas from the same source types used by the consumer where the provider supports that workflow, but still publish a canonical schema artifact and digest. “The SDK generated it” is not an identity.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness structured_output
cargo test -p mosaic-harness --example layered_structured_output
cargo run -q -p mosaic-harness --example layered_structured_output

Pinned output:

baseline                         accepted
mutation cases                  8
contract SHA-256                f85c49e87d9838e4eccffa39efac0e051777d858b2e445677d144dcf9d4d2ebc
generation SHA-256              7c2659c80644dd525309ee35ed1c067baf7faf823b8ff0514ee702d257d42981
validation-receipt SHA-256      c537573d3d5eaa75088d7582c7c28c546413864596e5ba0d1be5c80de6228e10
report bytes                    1,490
report SHA-256                  a987ccf3926e17abb72780f1698aabf588d08c32b0abf480c594ba7548ad913b

Thirteen module tests plus one complete-example test cover accepted replay, contract and decoder schema drift, required strict decoding, malformed JSON, unknown fields, accumulated semantic violations, unknown evidence/locators, typed refusal, truncation, pre-parse byte limits, evidence manifest mutation, strict contract decoding, and stored-receipt mutation.

Three mechanisms that are often confused

Prompted JSON

The instruction says “return JSON.” The decoder can still emit prose, Markdown fences, malformed escapes, missing fields, or a partial object. This can be adequate for exploration; it is not a strong machine interface.

JSON mode

The inference system constrains output to syntactically valid JSON, but not necessarily to your object schema. {"ok": true} can be valid JSON and useless to a consumer expecting an incident assessment.

Schema-constrained decoding

At each generation step, the inference system permits only continuations compatible with a grammar compiled from the schema. OpenAI describes dynamically restricting valid next tokens based on the supplied schema. llama.cpp exposes GBNF and JSON-Schema-to-grammar paths for local models.

Constrained decoding is an inference-time guarantee with an implementation-specific supported schema subset. It is not merely validation after generation, and it is not semantic proof.

allowed by grammar  ⊇  structurally meaningful  ⊇  semantically valid  ⊇  evidence-supported

The sets may be equal for a trivial enum. They are far apart for an incident report.

Start with the consumer’s decision

Do not create a giant “AI response” object. Ask what deterministic code must decide next. The reference assessment contains:

one bounded summary;
one allowed severity;
zero or more individually identified claims;
observation versus inference kind;
confidence in integer basis points, avoiding non-finite floats;
exact evidence and locator citations;
an optional root cause that references existing claim IDs;
explicit unknown questions and which evidence was attempted.

This structure permits a UI, evaluator, or workflow to distinguish observed facts from inference and unresolved questions. It does not encode hidden reasoning. Do not request private chain of thought as a field; request concise, auditable claims and evidence.

Design a closed and bounded schema

A durable output schema should specify:

schema release;
required versus optional fields;
whether null differs from absence;
closed objects where unexpected properties are rejected;
array and string maxima;
integer domains and enums;
unique stable IDs where cross-references exist;
explicit unknown/refusal representation;
descriptions that state meaning without granting authority.

JSON Schema permits properties that are not required unless required names them. It also permits additional object properties by default unless the schema closes them. Provider subsets vary, so compile and test the exact schema against every selected route.

The Rust output types use #[serde(deny_unknown_fields)]:

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct IncidentAssessment {
    pub schema_version: u32,
    pub summary: String,
    pub severity: Severity,
    pub claims: Vec<AssessmentClaim>,
    pub root_cause: Option<RootCause>,
    pub unknowns: Vec<Unknown>,
}

Strict deserialization rejects unknown keys. Separate deterministic checks enforce maximum claims, text bytes, confidence bounds, unique IDs, root-cause references, and citation membership.

Avoid unbounded recursive structures, maps with arbitrary model-invented keys, and broad unions. They enlarge the language, complicate provider portability, increase worst-case output, and create more semantically meaningless but schema-valid paths.

Bind contract, schema, and evidence identities

The reference contract is:

pub struct StructuredOutputContract {
    pub schema_version: u32,
    pub validator_release: String,
    pub contract_id: String,
    pub behavior_sha256: String,
    pub json_schema_sha256: String,
    pub evidence_manifest_sha256: String,
    pub require_strict_decoding: bool,
    pub max_output_bytes: usize,
    pub max_claims: usize,
    pub max_text_bytes: usize,
    pub allowed_severities: Vec<Severity>,
    pub evidence: Vec<EvidenceEntry>,
}

The contract validates its exact evidence entries and recomputes their manifest digest. Its own stable digest becomes contract_sha256. The provider request must use that contract and exact schema. The returned generation declares both:

pub struct StructuredGeneration {
    pub contract_sha256: String,
    pub model_release: String,
    pub provider_request_id: String,
    pub decoding_mode: DecodingMode,
    pub decoder_schema_sha256: Option<String>,
    pub finish_reason: FinishReason,
    pub payload: Option<String>,
    pub refusal: Option<ModelRefusal>,
    // schema_version omitted here
}

This reference verifies the declared decoder mode and schema identity. A production provider adapter must construct those declarations from the actual request it sent, not model text. Record the provider/model release and capability snapshot from Chapter 4 as well. If a provider silently falls back from strict schema mode, fail or route explicitly; do not label the response strict.

Keep the envelope outside the payload

The provider controls request IDs, status, finish reason, usage, refusal fields, and filtering signals. The model-produced JSON must not be able to impersonate them.

The reference state table is:

Finish reason	Payload	Refusal	Result
`stop`	exactly one	none	validate candidate
`refusal`	none	exactly one	typed refused outcome
`length`	any prefix	none	reject incomplete
`content_filter`	any prefix	none	reject incomplete/filtered

A refusal is neither malformed JSON nor an accepted assessment. OpenAI’s current guide explicitly documents that refusals may not follow the supplied output schema and should be handled through the separate refusal field. Other providers use different envelopes; normalize them without erasing the original status.

Never parse a length-truncated prefix merely because it happens to be valid JSON. It may be a valid object missing later intended claims. Completion status is part of correctness.

Validate in layers

Layer order makes failures actionable and prevents expensive or misleading later checks.

1. Envelope

Before parsing:

validate bounded identities;
compare contract_sha256;
require the chosen decoding mode;
compare the decoder’s schema digest;
interpret finish/refusal state;
reject payloads over the byte budget.

The output-size check precedes allocation-heavy parsing. Network adapters should also bound response bodies while reading.

2. Syntax

Parse one JSON value. Record only stable location information needed for debugging. Do not strip Markdown fences, search for the first {, repair quotes with regexes, or accept trailing prose. Those heuristics turn an invalid protocol into an ambiguous one.

3. Structure

Deserialize into the strict Rust type. Missing fields, wrong types, invalid enum spellings, and unknown keys fail here. If Rust types and the published schema are generated independently, add a compatibility test with accepting and rejecting corpora.

4. Semantics

The reference checks:

supported schema version;
summary and statement bounds;
contract-allowed severity;
maximum claim count;
at least one claim or explicit unknown;
unique and valid claim IDs;
confidence in 0..=10_000;
at least one citation per factual claim;
no duplicate citation within a claim;
root-cause references to existing unique claims;
unique attempted evidence IDs.

These rules cannot all be expressed portably in the provider’s schema subset, and some are domain rules rather than shape rules.

5. Evidence

Every citation must join to one evidence ID and one locator declared in the exact bound manifest. Every “attempted evidence” ID must also exist. Membership proves the cited artifact was available; it does not yet prove entailment. Chapter 10 adds source range, temporal, policy, contradiction, and support validation.

Run semantic checks before evidence checks in this reference. A structurally valid object with bad IDs should receive semantic violations without noisy downstream join failures. After repair, it can reach evidence validation.

Typed violations are part of the protocol

A rejection contains ordered entries:

pub struct OutputViolation {
    pub stage: ValidationStage,
    pub path: String,
    pub code: String,
    pub message: String,
}

Codes drive metrics, tests, repair policy, and UI. Messages help a developer or bounded repair call. Paths identify the field. Do not branch on prose messages.

The eight example cases deliberately land at distinct boundaries:

Mutation	Expected outcome
unchanged strict response	`accepted`
unconstrained decoder	`strict_decoding_required`
malformed JSON	`invalid_json`
unexpected field	`schema_mismatch`
disallowed severity	`severity_not_allowed`
citation outside manifest	`unknown_evidence_id`
length finish	`incomplete_generation`
provider refusal	`refused`

When a mutation moves to another layer, treat that as a contract change and review it.

Constrained decoding for remote and local models

Remote APIs often accept a documented JSON Schema subset and return a provider-specific envelope. Local runtimes may compile JSON Schema into a grammar or accept grammar text directly.

llama.cpp documents that its JSON schema support is a subset, lists unsupported or partially supported keywords, and recommends inspecting conversion warnings and testing the resulting grammar. It also notes that a schema used only for grammar construction may not be visible to the model. Therefore:

publish one canonical schema;
lint it against each route’s supported subset;
compile a route-specific grammar as a versioned derived artifact;
hash schema, compiler release/options, tokenizer/model release, and grammar;
test an accepting/rejecting corpus with the actual runtime;
describe field meaning in the prompt when the runtime does not inject schema descriptions;
retain deterministic post-generation validation.

A decoder can make a schema impossible to violate yet still force a weak model into a poor allowed choice. Evaluate task accuracy and abstention, not only schema-conformance rate.

Repair without hiding defects

Constrained decoding should reduce syntax and shape repairs. Semantic or evidence violations may still justify one bounded repair attempt.

The repair request includes:

original contract identity;
original candidate identity;
exact violation codes and paths;
an instruction to change only invalid fields;
unchanged evidence manifest;
remaining call/output/time budget.

Stop when the repair budget is exhausted, the same normalized violation repeats, the output becomes less complete, the provider refuses, or a non-retryable envelope/evidence failure occurs. Never ask the model to invent a missing evidence ID. Chapter 13 implements progress and stopping policy in full.

Store both candidates and receipts. Reporting only the final valid object hides failure rate and repair cost.

Versioning and compatibility

Version at least:

Rust output type and serialization form;
canonical JSON Schema;
route-specific schema/grammar compiler;
validator release;
behavior contract;
evidence manifest;
model/provider/runtime release.

A schema change can be additive in JSON Schema yet breaking for strict consumers or unsupported by a provider. Adding an enum value can make older exhaustive Rust matches fail. Changing a field description can alter model behavior even when the accepted language is unchanged. Run output evals for every release.

Prefer explicit version fields and parallel decoders during migrations. Validate old stored outputs with their original validator, then migrate through a typed, tested transformation. Do not reinterpret history under the newest schema silently.

Failure investigations

Valid JSON crashed the consumer

Determine whether the request used JSON mode instead of strict schema mode, whether the schema allowed missing/additional fields, whether the adapter used a different schema digest, or whether the consumer skipped strict deserialization. Add the exact payload to the rejecting corpus.

Strict output contained a false citation

The grammar guaranteed only a string at that field. Trace the context/evidence manifest and evidence-layer receipt. Add membership and support checks; do not weaken the schema or retry until a plausible ID appears.

A local model loops inside a valid array

Inspect array maxima, token budget, grammar complexity, repetition behavior, and finish state. Bound the collection, simplify the schema, and treat length termination as incomplete. A valid prefix is not a complete answer.

Provider upgrade increased semantic failures

Compare model release, schema identity, decoder capability, prompt release, accepted-output rate, semantic violation distribution, refusal, truncation, latency, and output tokens. Replay the same fixture corpus. Roll back the route through the release mechanism rather than adding permissive parsing.

First request with a schema is slow

Some runtimes compile and cache grammars. Measure cold and warm latency separately. Cache only by the exact schema/compiler/runtime identity and bound cache memory.

Security and performance

Structured output is still untrusted input. Bound encoded bytes and collection sizes. Reject NULs and control-character policies appropriate to the domain. Avoid attacker-controlled dynamic keys. Keep evidence text out of property names, schema descriptions, and violation messages sent to logs.

Schema descriptions and keys can influence generation; review them as prompt surface. A malicious retrieved document must never change the schema or claim to be a provider refusal. Sign or otherwise protect high-impact contract and validation receipts according to the threat model.

Measure:

envelope, syntax, structure, semantic, and evidence failure rates;
refusal and incomplete rates;
first-pass and repaired acceptance;
exact-schema conformance by route;
semantic accuracy and evidence support;
schema compilation cold/warm time;
validation p50/p95 and allocations;
output bytes/tokens and repair cost;
violation distribution by contract/model release.

Constrained decoding changes the allowed token set and may affect latency or answer quality. Measure with the actual model, tokenizer, schema, and runtime rather than generalizing from another route.

Alternatives and trade-offs

Free-form prose: best when a person is the only consumer and format flexibility matters.
Delimiter or Markdown template: easy to inspect, weak for nested machine data.
JSON mode plus validation: portable fallback when strict schema decoding is unavailable; expect more structure failures.
Function/tool calling: appropriate when the output chooses an executable capability; it adds authorization and effect concerns addressed in Chapter 11.
Grammar other than JSON: useful for a compact DSL or local runtime, but requires a safe parser and explicit versioning.
Deterministic extractor: preferable when the value already exists in a known structured source; a model adds unnecessary uncertainty.

Exercises

Recall

Why is JSON mode weaker than schema-constrained decoding?
What does strict decoding prove, and what does it not prove?
Why bind the decoder’s schema digest?
Why is refusal outside the model payload?
Why reject a parseable length-truncated object?
What is the difference between evidence membership and evidence support?

Implementation

Generate a canonical JSON Schema for IncidentAssessment and add its checked artifact.
Add maximum unknowns and maximum citations per claim.
Add an explicit NoRootCause reason instead of ambiguous null.
Compile the schema for a local grammar runtime and bind compiler identity.
Add compatibility fixtures proving the schema and Serde type accept/reject the same corpus.
Add a one-repair controller that stops on repeated violation signatures.
Property-test that accepted outputs never reference an unknown claim or evidence ID.
Fuzz the syntax/shape boundary under a strict input-byte limit.

Failure laboratory

Inject Markdown fences, trailing prose, duplicate keys, unknown fields, deep nesting, oversized strings, empty claims/unknowns, repeated claim IDs, confidence 10,001, a root cause referencing a missing claim, a citation using a real evidence ID with a false locator, a forged refusal inside the payload, a strict-mode downgrade, schema digest drift, truncation, and stored receipt mutation. Predict the rejecting layer before running.

Design

Design a structured output for a multimodal content-safety review. Separate observations about specific image regions/audio intervals/video ranges from policy conclusions. Specify uncertainty, abstention, evidence locators, model/provider release, reviewer escalation, collection bounds, and what must remain outside the payload envelope.

Solution guidance

Use a tagged enum or explicit status object for “root cause established” versus “insufficient evidence,” not an overloaded nullable string. Keep claims independently identified so conclusions can cite them. Bound nested arrays at both schema and deterministic-validation layers.

For provider portability, define a conservative canonical subset and a capability test per route. A route compiler emits a derived grammar plus warnings; CI rejects unsupported-keyword warnings. The runtime adapter records the schema/grammar identity it actually sent.

Normalize violation signatures from (stage, path, code) rather than message text. A repair is making progress only when required violations disappear without new equally severe ones.

Check your understanding

Which component owns the schema?
Can model text prove strict decoding was active?
Does valid JSON imply the Rust type will deserialize?
Does successful deserialization imply semantic validity?
Can an accepted citation still be unsupported?
Which finish reasons may enter payload validation?
What is the maximum payload before parsing?
How would you reproduce one validation decision?
What must be versioned when compiling JSON Schema to a local grammar?
Which metrics reveal a model that is structurally perfect but semantically weak?

Completion checklist

Authoritative references

OpenAI, Structured model outputs
OpenAI, Introducing Structured Outputs in the API
JSON Schema, Objects, required properties, and additional properties
JSON Schema, Getting started step by step
llama.cpp, GBNF and JSON Schema to grammar guide
mosaic-harness::structured_output and layered_structured_output