Syntactic, Structural, Semantic, and Evidence Validation

A schema-valid answer can still be confidently wrong.

Chapter 9 proved that an output used the expected contract, completed normally, parsed as JSON, matched a strict type, obeyed local invariants, and referenced known evidence IDs. This chapter asks the harder questions:

• does each citation actually support its claim?
• did the observation occur in the relevant window?
• does it belong to an allowed system and tenant?
• is the cited content the exact content in the manifest?
• does any source contradict the claim?
• is a root cause corroborated rather than repeated from one origin?
• should severity be computed by software instead of guessed by the model?

The architecture is:

schema-valid IncidentAssessment
        + exact structured-output receipt
        + evidence observations and provenance
        + independent claim-support judgments
        + deterministic impact measurement
        + domain validation policy
                         │
                         ▼
                semantic/evidence gate
        identity → source → time → domain → support
            → contradiction → confidence → severity
            → root-cause corroboration
                         │
               accepted or typed violations
                         │
                reproducible receipt

The runnable implementation is mosaic-harness::semantic_validation; its example is layered_semantic_validation.

Learning objectives

By the end you will be able to:

• define verification, validation, and evidence support as different checks;
• consume only an already accepted structured-output receipt;
• bind domain policy to the exact structured contract and evidence manifest;
• preserve content, provenance, locator, service, and time identity for observations;
• require exactly one independent judgment for every claim/citation pair;
• distinguish support, contradiction, unrelated evidence, and missing coverage;
• derive severity from deterministic impact observations;
• cap confidence for inference independently of schema shape;
• require multiple source origins for a root-cause conclusion;
• prevent model-derived evidence from corroborating itself;
• emit stable, replayable violations and receipts;
• test boundary values, mutations, and verifier disagreement.

Prerequisites and dependency order

Complete Chapter 9 first. Its accepted output is the input here, not a substitute for this layer. Complete the evidence/provenance and retrieval chapters so observations have content-addressed sources and exact locators. Complete behavior, policy, and run-envelope chapters so the validation policy knows the task, tenant, incident window, and release.

In production, the following should come from independent boundaries:

• the model produces claims, uncertainty, and citation references;
• ingestion/retrieval supplies evidence identity and locators;
• telemetry or authoritative systems supply impact values;
• a verifier, deterministic rule, specialist model, or human supplies support judgments;
• product policy derives severity and required corroboration;
• the validator joins them.

Letting one model produce the claim, invent the citation, judge its support, set severity, and approve the result is self-attestation.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness semantic_validation
cargo test -p mosaic-harness --example layered_semantic_validation
cargo run -q -p mosaic-harness --example layered_semantic_validation

Pinned result:

baseline                            accepted
mutation cases                     9
policy SHA-256                     fd7a91b99ebac4748fc9079708b5a6af6a40ece0cc8ffb920970ae8a3c437e86
structured-receipt SHA-256         dc04cc9546fa0afdf21c1271bbfa51c50123b555da4acc1125d34298c466b0b4
semantic-receipt SHA-256           90729455460df4c54e344fda6d1e325906c70f6314d9ee9e2c38b4858d3d04c1
report bytes                       1,796
report SHA-256                     09bfd262f697de849d86bb34e64cdd53f00be563f0cbad615382d8a1986f621e

Twelve module tests and three example tests cover accepted replay, all severity thresholds, content, time, service and manifest drift, missing/unknown judgments, unrelated and contradicting evidence, deterministic severity, overconfident inference, root-cause source diversity, derived-only corroboration, contract/manifest drift, rejected upstream output, duplicate identities, receipt tamper, and stable report cases.

Verification versus validation

Use the terms explicitly:

• Syntactic verification: is the byte sequence one complete JSON value?
• Structural verification: does it instantiate the versioned type/schema?
• Semantic validation: do values and relationships make sense for this task and domain?
• Evidence validation: do exact sources support the claims under a declared method?
• Decision validation: does the resulting action satisfy policy and risk tolerance?

The last item may happen after this chapter. A supported diagnosis does not automatically authorize a rollback. Tool effect authorization belongs to Chapter 11.

NIST uses testing, evaluation, verification, and validation as a broader operational discipline and emphasizes objective, repeatable, documented processes. Treat this implementation as one product control inside that discipline, not a certificate of universal truth.

Bind validation policy to upstream identity

The reference policy includes:

pub struct SemanticValidationPolicy {
    pub structured_contract_sha256: String,
    pub evidence_manifest_sha256: String,
    pub incident_start_unix_ms: u64,
    pub incident_end_unix_ms: u64,
    pub allowed_services: Vec<String>,
    pub maximum_inference_confidence_bps: u16,
    pub minimum_root_cause_sources: usize,
    pub thresholds: SeverityThresholds,
    // schema_version and validator_release omitted
}

It checks both the contract passed to validation and the contract identity stored in the structured receipt. It also checks the evidence-manifest identity declared by that contract.

Without these joins, a valid claim from run A can be validated under run B’s looser policy, or a citation can be resolved against a newer index generation. Stable digests are not security by themselves; they make the object being signed, authorized, traced, and reproduced unambiguous.

Accept only the upstream accepted state

validate_semantics_and_evidence refuses to run when Chapter 9 returned Refused or Rejected. There is no meaningful evidence validation for malformed output.

A production caller should first replay the structured receipt from the original contract and generation. This reference binds its receipt identity as an input. Store the layered receipts together:

generation
  └── structured validation receipt
       └── semantic/evidence validation receipt
            └── behavior-completion receipt

Each layer answers a narrower question and names its dependencies.

Turn citations into observations

The structured output contains only citation references. The independent evidence plane supplies:

pub struct EvidenceObservation {
    pub evidence_id: String,
    pub locator_id: String,
    pub content_sha256: String,
    pub provenance_sha256: String,
    pub service: String,
    pub observed_start_unix_ms: u64,
    pub observed_end_unix_ms: u64,
    pub origin: ObservationOrigin,
}

The validator checks:

1. evidence ID exists in the bound manifest;
2. content digest matches the manifest entry;
3. locator is declared for that evidence;
4. observation is inside the half-open incident window;
5. service belongs to the policy allowlist;
6. (evidence_id, locator_id) identity is unique.

Real observations should also bind tenant, acquisition time, clock map, source classification, retrieval snapshot, and access-policy decision. Multimodal observations use source byte ranges, image regions, audio intervals, video ranges, and transformation maps from Part 4.

Do not broaden a citation locator during validation. If a claim cites lines 10–14, validating the whole file may hide a false local attribution behind generally relevant content.

Require exact judgment coverage

One judgment addresses one claim/citation pair:

pub struct ClaimSupportJudgment {
    pub claim_id: String,
    pub evidence_id: String,
    pub locator_id: String,
    pub relation: SupportRelation,
    pub verifier_release: String,
    pub judgment_sha256: String,
}

pub enum SupportRelation {
    Supports,
    Contradicts,
    Unrelated,
}

The gate constructs the complete set of citation triples from the accepted output and requires one judgment for each, with no extra triple. Missing coverage and unknown judgment are different:

• missing coverage means verification did not finish;
• unknown judgment means a verifier evaluated something the model did not cite;
• unrelated means the source does not establish this claim;
• contradicts means the source supplies opposing evidence.

The reference accepts a claim only when every citation has a judgment, at least one supports it, no citation is unrelated, and none contradicts it. Products may allow a mixture of supporting and unrelated background citations, but must then measure citation precision separately and prevent irrelevant citations from appearing as support.

Citation evaluation research distinguishes whether a cited passage supports the claim from whether all claims are covered. Both matter. Automatic entailment or LLM judges are fallible; calibrate them against human labels and retain an escalation band.

Keep verification independent

An “independent verifier” means it does not merely repeat the producer’s assertion. Useful options:

• deterministic checks against typed telemetry;
• database or control-plane reads;
• domain-specific rules;
• a specialist natural-language-inference model;
• a different prompt/model with hidden expected criteria;
• human review for high-impact or ambiguous claims.

Independence is not guaranteed merely by using a second call to the same model. Shared training, prompt, retrieved context, or systematic bias can correlate errors. Record verifier release and measure disagreement by claim type, source type, domain, and difficulty.

For important facts, sample human audits even when the automatic verifier agrees. Freeze verifier test sets and prevent optimizer access.

Derive policy values deterministically

The model may describe impact; software should compute policy categories from authoritative measurements where possible.

The example severity function is:

critical if confirmed data loss or affected users ≥ 10,000
high     if failed requests ≥ 1,000 or affected users ≥ 1,000
medium   if failed requests ≥ 100 or affected users ≥ 100
low      otherwise

Thresholds live in the policy, not the prompt. Boundary tests prove 99/100 and 999/1,000 behavior. The model’s declared severity must equal the derived value.

This pattern applies beyond incidents:

• calculate monetary totals instead of trusting model arithmetic;
• derive access class from policy labels;
• compute date ranges with a time library;
• enforce media geometry against decoded dimensions;
• look up deployment state from the control plane;
• calculate confidence gates from calibrated scores.

Use the model for interpretation where deterministic computation is unavailable, not where reliable software already owns the fact.

Cap inference confidence

Confidence is a claim about uncertainty, not decoration. The schema permits 0..=10_000 basis points. Domain policy can impose a lower maximum on inferred claims until calibration supports more. The fixture caps inference at 8,500 while allowing a direct observation at 9,800.

This is a coarse safety rule, not calibration. Chapter 6’s calibration material and the evaluation part of the book measure whether reported probabilities correspond to observed frequencies. Slice by claim kind and verifier agreement. If values are not calibrated, use ordinal confidence or explicit evidence quality instead of pseudo-precise probabilities.

Root cause needs corroboration

A root cause is stronger than a correlated observation. The reference requires:

• root-cause support references existing claims;
• supporting citations receive support judgments;
• distinct evidence IDs meet a policy minimum;
• at least one supporting observation is not model-derived.

Two summaries of one log are one origin, not two sources. A model-generated hypothesis retrieved from memory cannot validate the same hypothesis. Track origin lineage and collapse derivatives to their roots when counting corroboration.

Distinct sources can still share a failure mode. Logs and traces emitted by the same faulty instrument may be correlated. For high impact, require heterogeneous evidence—such as telemetry, deployment history, and a controlled reproduction—or a human decision.

The example’s root cause uses one API log and one database trace. Raising the required count from two to three rejects it. Marking both observations model-derived also rejects it.

Contradictions are first-class

Do not average contradictory judgments into a high confidence score. A contradiction produces a typed violation and routes to investigation.

Possible resolutions:

• claim scope is too broad;
• sources describe different time windows or tenants;
• one observation is stale or corrupted;
• the model conflated correlation and cause;
• verifier disagreement needs adjudication;
• the domain genuinely changed over time.

Preserve opposing evidence in the review case. The repair request may narrow the claim or move it to unknown; it must not omit the contradicting source simply to pass.

Failure investigations

A citation ID exists but does not support the sentence

Inspect the exact locator, source bytes, transformation lineage, claim boundaries, support-judgment release, and human audit label. Split compound claims so every atomic proposition can be judged. Add the pair to verifier evaluation.

Correct evidence was rejected as outside the incident

Compare source clock, canonical clock map, timezone, inclusive/exclusive boundary, and acquisition versus event time. Repair clock mapping or policy. Do not widen the incident window globally.

Severity changes between identical runs

Check impact measurement identity, policy thresholds, event snapshot, integer overflow, and ordering. Severity must not come from free-form model text or current mutable dashboards during replay.

Root cause passed with two copied summaries

Trace provenance to root sources. Count independent roots, not artifact IDs. Add a mutation in which two derivatives share one origin and require rejection.

Verifier agrees in tests and fails in production

Compare domain/source/claim distributions, truncated locators, prompt/model release, calibration, and human disagreement. Create a production-failure slice and lower automatic authority until the verifier is requalified.

Security and performance

Support judgments are security-sensitive. An attacker may inject a fake judgment, choose an overbroad locator, duplicate one origin, shift timestamps, exploit a stale policy, or make verifier content contain instructions. Treat source and judgment text as data. Authenticate verifier identity, protect receipts, and authorize access before the verifier sees evidence.

Bound observations and judgments before building maps. The reference uses ordered maps/sets for deterministic decisions. For large outputs, use keyed joins while retaining canonical sorting for identity. Cache immutable support judgments only by exact claim, source bytes, locator, transformation, verifier, and policy identities.

Measure:

• claim coverage and citation precision;
• support, unrelated, contradiction, and missing rates;
• automatic/human agreement and calibration;
• wrong-time, wrong-service, digest, and manifest failures;
• severity disagreement;
• root-cause source count and origin diversity;
• p50/p95 validation latency and verifier cost;
• repair/escalation rate;
• false acceptance and false rejection on adjudicated sets.

Alternatives and trade-offs

• Human verification for every output: strongest review for low volume, costly and still dependent on source presentation and reviewer consistency.
• Deterministic-only claims: excellent where structured systems expose all facts, insufficient for ambiguous unstructured evidence.
• Single LLM judge: cheap and broad, but correlated and calibratable rather than authoritative.
• NLI specialist: efficient for passage/claim support, domain shift and compound claims remain risks.
• No root-cause claim: often the right product choice; report observations and unknowns until evidence is sufficient.

Exercises

Recall

1. What does schema validity fail to prove?
2. Why is citation membership weaker than support?
3. How do missing, unrelated, and contradictory judgments differ?
4. Why derive severity outside the model?
5. Why do two derivatives of one source not corroborate?
6. Why can a second model call still be correlated with the producer?

Implementation

1. Bind tenant and retrieval-snapshot identity to every observation.
2. Count provenance roots rather than evidence IDs for corroboration.
3. Add PartialSupport and define compound-claim behavior.
4. Add an adjudication object for verifier disagreement.
5. Require heterogeneous source categories for critical root causes.
6. Add half-open temporal boundary property tests.
7. Add calibrated support scores with an escalation band.
8. Add a deletion/revocation check so invalidated evidence cannot support output.

Failure laboratory

Mutate accepted/rejected upstream state, contract digest, manifest digest, evidence bytes, locator, tenant, service, time start/end, duplicate observation, missing/extra judgment, support to unrelated, support to contradiction, inference confidence, impact threshold, root source count, and provenance origin. Predict malformed input versus valid rejection and the exact violation.

Design

Design semantic/evidence validation for a generated image-edit plan. Validate source asset and region identity, pixel bounds, edit type, consent for human likeness, non-destructive output path, expected visual result, policy severity, and evidence that the rendered artifact actually reflects the plan. Separate proposal validation from authorization to execute.

Solution guidance

Represent claim support as a join over (claim_id, evidence_id, locator_id). Keep duplicate triples malformed rather than silently choosing one. Compute complete coverage before interpreting relations.

For corroboration, traverse provenance to acquired roots and count unique roots by category. A derived OCR block and image caption from one screenshot remain one acquired source.

For verifier disagreement, retain every signed judgment, calculate policy from an adjudicated result, and store the adjudicator release. Do not overwrite the losing judgment.

Check your understanding

• Which upstream state may enter this validator?
• What binds the validation policy to the structured result?
• Which evidence fields must match the manifest?
• Are acquisition time and event time interchangeable?
• Can one supporting citation offset a contradicting citation?
• Which values should software derive?
• What prevents a model-derived hypothesis from supporting itself?
• How is root-source diversity measured?
• What must a cached support judgment key contain?
• Which metrics reveal silent false acceptance?

Completion checklist

• Only an accepted structured output enters.
• Policy binds exact structured contract and evidence manifest.
• Observations retain content, provenance, locator, service, and time.
• Tenant/policy/retrieval identity is added in production.
• Every citation triple has exactly one independent judgment.
• Missing, unknown, unrelated, and contradictory states remain distinct.
• Deterministic values are derived from authoritative observations.
• Inference confidence has an explicit policy and calibration plan.
• Root cause requires sufficient independent origins.
• Model-derived evidence cannot self-corroborate.
• Contradictions route to investigation, not averaging.
• Decisions and receipts reproduce from independent inputs.
• Boundary, mutation, property, disagreement, and replay tests pass.
• Human audits calibrate automatic support validation.
• Security, latency, cost, and false-decision metrics are monitored.

Authoritative references

• NIST, AI Risk Management Framework Core: Measure
• NIST, AI test, evaluation, validation and verification
• Gao et al., Enabling Large Language Models to Generate Text with Citations
• Yue et al., Automatic Evaluation of Attribution by Large Language Models
• Aly et al., Learning to Generate Answers with Citations via Factual Consistency Models
• mosaic-harness::semantic_validation and layered_semantic_validation