Retrieval Contracts, Snapshots, and Reranking

Retrieval is not “put the nearest chunks in the prompt.” It is a bounded evidence-selection pipeline. The pipeline answers four different questions:

Which corpus, permissions, index releases, and query representation did this run use?
Which candidates did each retrieval channel return?
How did fusion, reranking, and diversity change their order?
Which exact artifacts became evidence candidates for context allocation?

A generated answer can be wrong even when the model behaved perfectly. The relevant record may have been absent from the index, excluded by a tenant filter, missed by approximate search, removed by reranking, collapsed as a near-duplicate, or dropped by the context budgeter. Calling every one of those failures “the model hallucinated” makes the system impossible to improve.

This chapter builds mosaic-harness::retrieval_contract. It complements the ranking algorithms in mosaic-evidence::retrieval: the evidence crate implements deterministic weighted reciprocal-rank fusion, complete reranker judgments, bounded maximal-marginal-relevance selection, and source-restorable citations; the harness crate binds those computations to one versioned request, four index snapshots, one tenant policy, and a reproducible receipt.

query artifact + tenant policy
               │
               ▼
       versioned retrieval plan
        │       │       │       │
     lexical semantic structured multimodal
        │       │       │       │
        └────── channel receipts ──────┐
                                      ▼
                    fusion → rerank → diversity
                                      │
                               final receipt
                                      │
                         untrusted evidence candidates
                                      │
                         MP-21 context allocation

The important boundary is the last arrow. Retrieval proposes evidence. It does not decide that the evidence outranks instructions, recent user input, tool schemas, memory, or the promised output reserve.

Learning objectives

By the end you will be able to:

separate lexical, semantic, structured, and multimodal retrieval responsibilities;
explain why scores from different retrievers cannot usually be compared directly;
bind a request to immutable index, corpus, schema, tenant, and policy identities;
apply authorization and deterministic filters before approximate candidate generation;
preserve one artifact identity when the same candidate appears in several channels;
validate per-channel top-k, threshold, rank, and ordering contracts;
fuse rankings without pretending their native score scales are calibrated;
rerank a complete declared pool and preserve the earlier-stage receipts;
use diversity selection without erasing evidence coverage;
retrieve derived media representations while retaining a locator to the original;
turn final hits into explicitly untrusted context candidates;
reproduce a receipt and detect stale snapshots, identity drift, and stored-result mutation;
measure retrieval recall separately from answer correctness;
debug zero-result, irrelevant-result, duplicate-result, and cross-tenant failure modes.

Prerequisites and dependency order

Complete multimodal acquisition, provenance, chunking, embeddings, and ranking before this chapter. Complete the behavior contract, run envelope, prompt release, provider-neutral request, and context budget chapters from Part 5.

The dependency order is deliberate:

ingestion defines what can be found;
provenance defines what can be trusted and cited;
the run envelope supplies principal, tenant, policy, and budget;
retrieval produces ranked evidence candidates;
the context allocator decides which candidates fit;
the provider compiler renders the selected content;
evaluation attributes failure to the correct stage.

Do not let a retriever inspect the entire context window and silently truncate everything else. Do not let a model invent a search query before the run contract has authorized the corpus. Do not let the provider adapter become the only place where retrieved evidence identities exist.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness retrieval_contract
cargo test -p mosaic-harness --example snapshot_bound_retrieval
cargo run -q -p mosaic-harness --example snapshot_bound_retrieval

The reference executes seventeen contract tests and one complete example test. Its pinned result is:

channels                         lexical, semantic, structured, multimodal
snapshots                        4
final candidates                 log, trace, screen
context candidates               retrieval:log, retrieval:trace, retrieval:screen
plan SHA-256                     5bfa8288fe9d3dbd95e89c3e9eb0ce12a0dc5c4884d930b80d6a96be58612250
channel-results SHA-256          707e97c17d774cf5b3f94b7baad7a49f124ce1a73c6ec162e33e8c7cf979beb1
receipt SHA-256                  e111d7e022513c1ee85aecef50eb3a7b68bd42f1989283dd87a46e59e850cb7d

The example answers “why did the ingest job fail?” Exact identifiers favor the runbook and log, semantic similarity recovers the log and trace, structured retrieval finds the high-signal trace row, and multimodal retrieval finds a screenshot. The declared downstream stages select the log, trace, and screenshot. The final handoff marks all three as untrusted evidence in the context allocator’s evidence lane.

Start with a retrieval plan, not a vector call

The reference plan contains:

pub struct RetrievalPlan {
    pub schema_version: u32,
    pub contract_release: String,
    pub request_id: String,
    pub tenant_id: String,
    pub policy_sha256: String,
    pub query_sha256: String,
    pub query_modality: RetrievalModality,
    pub snapshots: Vec<IndexSnapshot>,
    pub channel_policies: Vec<ChannelPolicy>,
    pub fusion_release: String,
    pub reranker_release: String,
    pub diversity_release: String,
    pub final_limit: usize,
}

This is intentionally more explicit than:

search(query, top_k)

The short interface hides every input that changes behavior. top_k = 10 means little without the corpus, index build, embedding model, query transformation, filters, permission snapshot, distance function, approximate-search parameters, fusion rule, reranker, and diversity rule.

The plan binds the request to one query artifact hash rather than storing only a display string. That query artifact can include:

normalized user text;
exact identifiers extracted without a model;
a model-generated rewrite plus its prompt/model/trace identity;
an image or audio query embedding input;
structured predicates;
time and modality constraints.

Hash the complete query representation. If a rewrite changes, it is a new retrieval input even if the original user message is identical.

Bind every channel to an immutable snapshot

Each IndexSnapshot records:

pub struct IndexSnapshot {
    pub channel: RetrievalChannel,
    pub snapshot_id: String,
    pub retriever_release: String,
    pub schema_sha256: String,
    pub corpus_sha256: String,
    pub tenant_id: String,
    pub policy_sha256: String,
}

The snapshot ID is not enough by itself. A mutable alias such as production-latest can point to different bytes tomorrow. The corpus digest binds the indexed evidence set; the schema digest binds tokenization, fields, dimensions, distance semantics, metadata representation, and locator shape; the retriever release binds query-time logic.

The tenant and policy identities close a serious security gap. A correct vector match from the wrong tenant is still a data breach. A record allowed by yesterday’s policy may be forbidden today. The reference refuses to construct a plan when any channel snapshot names a different tenant or policy.

In a production store, the digest may identify a signed manifest rather than hashing a multi-terabyte index directly. The manifest should cover:

every segment, shard, or table snapshot;
source-ingestion cutoffs;
tombstones and deletion watermarks;
embedding, tokenizer, parser, OCR, and media-transform releases;
index parameters;
permission-materialization version;
checksums for all referenced artifacts.

Keep old manifests and the ability to query or reconstruct them for the replay horizon promised by the product. “We know which index existed” is not reproduction if the index has already been destroyed.

Give the four channels different jobs

Lexical retrieval

Lexical search is strong when the query contains exact surface forms:

symbol and function names;
error codes;
filenames and paths;
product SKUs;
quoted phrases;
identifiers with rare character sequences.

BM25-family ranking combines term rarity, term frequency saturation, and document-length normalization. Lucene’s current BM25Similarity documentation exposes the ranking implementation and parameters rather than treating “keyword search” as a boolean feature. Use the scorer’s native value to order that lexical list, but do not assume it has the same meaning as cosine similarity.

Tokenization is part of the index schema. Lowercasing, stemming, compound splitting, Unicode normalization, punctuation handling, and field boosts can decide whether an identifier is retrievable. Test exact domain strings, not only natural-language questions.

Semantic retrieval

Semantic retrieval maps queries and candidate representations into a vector space. It is useful for paraphrase, conceptual similarity, and cross-modal alignment. Its failure modes include:

embedding-model drift;
domain language outside the training distribution;
truncation before the decisive sentence;
hubness and near-duplicate clusters;
wrong distance function or normalization;
approximate-index recall loss;
metadata filtering that removes neighbors after the scan.

The official pgvector documentation distinguishes exact nearest-neighbor search from approximate HNSW and IVFFlat indexes. Approximate indexes trade recall for speed. It also documents a subtle filtering problem: filtering after an approximate scan can return too few rows, so iterative scans, partial indexes, partitioning, or exact search may be required. That is a retrieval-system decision, not a prompt problem.

Measure approximate recall against exact search on a representative sample:

recall@k = |approximate_top_k ∩ exact_top_k| / k

Break the metric down by tenant size, filter selectivity, modality, language, and corpus age. A single global average can hide a complete failure for a small tenant or rare modality.

Structured retrieval

Structured retrieval uses typed fields and relationships:

run_id = ...;
status = failed;
event time within a range;
ownership or service equality;
graph edges;
numeric thresholds;
exact artifact or policy identities.

When a user asks why one known run failed, a primary-key or indexed predicate is more reliable than embedding the run ID. Apply authorization inside the structured query. Never retrieve broadly and ask the model to ignore unauthorized rows.

Structured results still need provenance and locators. A database row can change. Record the table or collection snapshot, primary key, row version, relevant column projection, and a stable artifact digest. If an aggregation produced the evidence, bind the query plan and input snapshot.

Multimodal retrieval

Multimodal retrieval may search:

OCR text from images;
captions and region embeddings;
audio transcripts and acoustic embeddings;
shot, frame, or clip embeddings;
layout blocks and table structures;
cross-modal image-text or audio-text spaces.

Search derived representations, but cite the original. A screenshot found through OCR should retain the image artifact digest and region locator. A video segment found through a transcript should retain the original video, exact time range, named clock, and clock-map identity. This is why ArtifactIdentity contains content, provenance, and locator digests.

Do not merge every modality into one anonymous embedding table unless you can still enforce modality-specific validity, access, retention, and locator rules.

Filter before approximate retrieval

A safe order is:

principal + tenant + policy
        │
        ├─ deterministic authorization scope
        ├─ deletion/retention state
        ├─ time and modality constraints
        └─ corpus snapshot
                    │
             candidate generation

Pre-filtering reduces the chance of unauthorized material affecting scores, caches, timing, or model input. When an index engine applies metadata filters after approximate candidate generation, understand the consequence. You may need a tenant-specific partition, a partial index, a larger scan, or an exact fallback.

Treat zero results as an observable outcome with reasons:

no authorized records matched;
authorized corpus was empty;
query representation failed;
threshold removed all hits;
approximate scan exhausted before enough filtered hits;
modality was unavailable;
index snapshot was unhealthy.

Returning the same empty vector for all cases destroys operability.

Keep score spaces separate

The harness accepts signed fixed-point native scores within a bounded diagnostic range. Integers avoid NaN, infinity, platform-dependent serialization, and accidental JSON changes. The score is still channel-local.

Suppose:

lexical BM25       17.2
cosine similarity  0.81
SQL rule score     1.00
image-text score   0.29

Sorting those four raw numbers together is meaningless. Even two cosine values may not be comparable when they come from different embedding releases.

There are three defensible approaches:

calibrate each score into a common probability-like target using held-out judgments;
train a fusion model with versioned features and evaluation;
fuse ranks when score calibration is unavailable.

The companion evidence crate uses weighted reciprocal-rank fusion (RRF). For candidate (d):

RRF(d) = Σ_i weight_i / (k + rank_i(d))

RRF combines ordering rather than raw score magnitude. The original Cormack, Clarke, and Buettcher paper reports that this simple method performed robustly against individual rankers in their experiments. That is evidence for evaluating RRF, not permission to copy one constant into every product. Tune weights and the reciprocal-rank constant on development data, then freeze them as a release.

Rerank the declared pool completely

A richer reranker can consider the query and candidate together. It may be a cross-encoder, a small language model, a deterministic domain rule, or a learned multimodal model.

The contract must name:

the exact fused-pool identity;
the reranker model and tokenizer release;
input truncation policy;
feature schema;
score interpretation;
batch and numerical policy;
one judgment for every candidate in the declared pool.

Partial reranker output is not silently “good enough.” If the service scores 17 of 20 candidates and the harness keeps the missing three at arbitrary old positions, a timeout has changed behavior without a receipt. Choose and record a fallback: fail the stage, use fusion order for the whole pool, or invoke a separately evaluated degraded route.

Never ask a generative model to return candidate IDs and accept invented or duplicated values. Join results against the exact input set and require complete cardinality.

Select diversity without erasing coverage

Top-ranked candidates are often redundant: repeated log lines, near-identical documentation chunks, adjacent video frames, or several crops of the same screenshot. Maximal marginal relevance (MMR) is a greedy way to trade relevance against similarity to already selected items:

choose argmax_d [
  λ × relevance(d, query)
  - (1 - λ) × max_similarity(d, selected)
]

The companion uses a bounded, complete pair-similarity matrix so missing similarities cannot silently behave like zero. Candidate IDs break exact ties deterministically.

Diversity is not always the same as coverage. A debugging task may require three adjacent events to establish causal order. A compliance task may require every mandated source even when two are similar. Add coverage groups or required evidence separately; do not expect one novelty coefficient to encode product semantics.

Preserve candidate identity across every stage

The same logical candidate can appear in lexical and semantic results. The reference requires its ArtifactIdentity to match exactly:

pub struct ArtifactIdentity {
    pub content_sha256: String,
    pub provenance_sha256: String,
    pub locator_sha256: String,
}

If log refers to one content digest in lexical search and another in semantic search, the pipeline fails. It does not guess that they are “probably the same.” Common causes include:

one index updated before another;
a reused candidate ID;
a chunk re-created with different boundaries;
a mutable URL whose bytes changed;
locator drift after media transcoding;
tenant-local IDs incorrectly treated as global.

Use globally scoped compound identity or keep tenant and snapshot explicit in every join.

Validate channel results before fusion

Each channel result declares the snapshot ID it actually queried. The validator requires:

exactly one result for every planned channel;
exact snapshot equality;
result count no greater than declared top_k;
contiguous ranks beginning at one;
descending native scores;
every score at or above the channel threshold;
unique candidate IDs within the channel;
bounded IDs, scores, token counts, and counter releases;
valid content, provenance, and locator digests.

A “top-k” contract is a maximum, not a promise that every channel returns k items. Empty results are allowed when genuine. But missing channel execution is different from an executed empty result and should have a typed stage outcome in a production trace.

The reference requires all four channels because the exercise demonstrates a full mixed pipeline. A general product contract can support an explicit enabled-channel set. Do not infer enablement from which result arrays happen to be present.

Validate the final stage

FinalHit records:

candidate and artifact identity;
modality;
contiguous final rank;
fused score units;
reranker score;
token count and counter release.

The validator joins each final hit back to the earlier channel results. Unknown candidates, duplicate candidates, identity changes, modality changes, token-count changes, and noncontiguous ranks fail. This prevents a downstream stage from smuggling in material that was never authorized and retrieved.

The receipt then hashes:

the complete plan;
the ordered channel results;
each channel result separately;
the ordered final hits.

verify_retrieval_receipt recomputes the receipt from the independently supplied plan, channel results, and final hits. It catches stored-receipt mutation instead of trusting the stored final values as their own source of truth.

Hand verified hits to the context allocator

evidence_context_candidates converts final hits into MP-21 candidates:

lane                  evidence
trust                 untrusted_evidence
placement             variable_body
required              false
priority              final-rank-derived
utility               deterministic rank utility
content identity      retrieved artifact content digest
provenance identity   retrieved provenance digest
token receipt         exact content + counter release

“Untrusted” is not the same as “incorrect.” It means retrieved content cannot acquire instruction authority merely because it resembles a relevant document. A web page that says “ignore prior instructions” remains evidence data.

The simple rank-derived utility in this chapter demonstrates a deterministic boundary. A production policy should derive utility from evaluated task coverage, trust, novelty, recency, and marginal answer value. Keep that policy versioned. Retrieval rank alone does not know whether an item is worth displacing recent conversation or a required tool declaration.

The context allocator may drop a correctly retrieved candidate because of lane or total budget. That decision belongs in the context manifest, while the retrieval receipt proves the candidate was available. This separation makes failure attribution possible.

Failure investigations

Failure: relevant evidence exists but is never returned

Investigate in order:

Verify the source artifact and ingestion lineage.
Confirm the exact corpus snapshot includes it.
Test deterministic tenant, policy, time, and modality filters.
Inspect chunk boundaries and derived representations.
Run the channel query against exact search where possible.
Compare approximate and exact top-k.
Inspect query rewriting and embedding truncation.
Check thresholds before blaming fusion.

Add the case to a retrieval evaluation set. A one-off manual query is diagnosis, not regression protection.

Failure: cross-tenant result appears

Stop the route and treat it as a security incident. Do not add a prompt telling the model not to mention the result. Inspect authorization placement, cache keys, shared index filters, snapshot tenant identity, and candidate-ID scoping. Add a negative test that uses semantically identical records in two tenants.

Failure: ten results say the same thing

Inspect source duplication, overlapping chunks, fusion contributions, and pairwise similarity. Deduplicate exact content before expensive reranking. Use lineage-aware diversity so multiple derived artifacts from one source do not crowd out independent evidence. Then measure whether diversity improves answer coverage rather than only visual variety.

Failure: receipt cannot reproduce after a deploy

Diff the plan and stage identities before diffing model output:

query → tenant/policy → corpus/schema → retriever → result lists
      → fusion → reranker → diversity → final hits → context manifest

If an old index cannot be queried, reconstruct from its signed corpus manifest or mark the run non-replayable with the exact missing dependency. Never substitute “latest” and call it replay.

Failure: reranker improves offline metrics but hurts production

Check pool recall, truncation, latency timeouts, missing-judgment fallbacks, score calibration, language/modality slices, and selection bias in the judged dataset. A reranker cannot recover a candidate absent from its input pool. Report candidate-generation recall and reranker quality separately.

Security implications

Retrieval systems expose more than returned text. Timing, hit counts, cache behavior, error differences, embeddings, and generated query rewrites may reveal corpus membership.

Apply these controls:

authorize before material crosses a tenant boundary;
include tenant, policy, corpus, model, transform, and query identities in cache keys;
do not log raw sensitive queries or content by default;
encrypt indexes and artifacts according to their source classification;
propagate deletion and revocation into every derived index;
prevent retrieved content from gaining instruction authority;
validate media parsers and bound decompression before indexing;
rate-limit expensive or adversarial high-recall queries;
isolate evaluator and development corpora from production memory;
audit model-generated query expansion for data exfiltration.

If embeddings are sensitive in the product’s threat model, treat them as derived confidential data, not anonymized text.

Performance implications

Budget retrieval as a staged funnel:

cheap filters
  → bounded channel candidate generation
  → rank fusion
  → expensive rerank of a small pool
  → bounded diversity selection
  → context allocation

Record latency, scanned candidates, returned candidates, cache outcomes, and stage budgets. Set a deadline for each stage and define the fallback before production.

Useful measurements include:

exact and approximate recall@k;
judged precision@k and nDCG;
citation/source recall;
duplicate rate and source coverage;
reranker pool size and complete-judgment rate;
context inclusion rate for retrieved relevant evidence;
p50/p95/p99 stage latency by corpus and filter selectivity;
index freshness and deletion lag;
query and candidate tokenization cost;
cross-modal transform and accelerator utilization.

pgvector’s documentation recommends comparing approximate results with exact search and using EXPLAIN (ANALYZE, BUFFERS) for query diagnosis. Equivalent observability is needed whichever engine you use.

Alternatives and trade-offs

One hybrid search engine

A single engine can reduce operational joins and may provide built-in lexical/vector fusion. Still record independent channel and snapshot semantics. Product behavior should not become uninspectable because one vendor returns a final score.

Application-side fusion

Application-side fusion offers explicit contracts, custom weighting, and engine independence. It adds network calls, result joining, and consistency obligations. The reference design is suitable when reproducibility and mixed evidence sources matter more than minimal plumbing.

Learned fusion

A learned ranker can outperform fixed RRF when representative judgments and stable features exist. It introduces training-data governance, calibration, feature drift, serving, rollback, and explanation requirements. Keep the deterministic fusion route as an evaluated fallback.

Generative retrieval and query planning

A model can decompose questions, generate structured predicates, or perform iterative search. Treat each generated query as a bounded tool call with exact prompt/model/output identities. Authorize each query, cap fan-out, detect repeated plans, and preserve every intermediate receipt.

Exercises

Recall

Why can BM25 and cosine scores not be sorted together directly?
What is the difference between an empty channel result and a missing channel execution?
Which identities make an index result replayable?
Why is retrieved evidence marked untrusted?
Why can a reranker not repair low candidate-generation recall?
How can metadata filtering reduce approximate-search recall?

Implementation

Add an explicit enabled-channel set while still distinguishing missing from empty results.
Add a FilterReceipt containing the typed predicate, authorization decision, and before/after candidate counts. Reject predicates containing unbound tenant fields.
Add a complete fusion-stage receipt that joins every contribution to channel rank.
Connect mosaic-evidence::fuse_ranked_lists directly to this contract through a typed adapter.
Add a fallback enum for reranker timeout: Fail, UseFusionOrder, or UseNamedRoute.
Add property tests that permute result construction order but preserve the canonical receipt.
Build an exact-versus-approximate recall fixture with deliberately selective metadata filters.
Extend the example with a video hit whose locator binds a time range and clock map.

Failure laboratory

Create mutations for:

a stale snapshot alias;
the same candidate ID with different bytes across channels;
a rank gap;
an ascending native score;
a channel exceeding top-k;
a final hit absent from every channel;
a locator change after reranking;
a tenant mismatch in one index;
a stored receipt with one changed fusion unit;
a retrieved prompt-injection string.

The first nine must fail closed at the contract boundary. The final string should remain accepted as untrusted evidence data; instruction authority must not change.

Design

Design retrieval for a support system containing tickets, logs, screenshots, call audio, runbooks, and account records. Specify:

channel and corpus boundaries;
tenant and policy enforcement;
source and derived artifact identities;
exact and approximate indexes;
query planning;
fusion and reranking;
diversity and required coverage;
context utility;
replay horizon;
security and deletion behavior;
offline and online evaluation slices.

Explain the behavior when one channel is unavailable and when a policy changes between ingestion and query time.

Solution guidance

For enabled channels, store a sorted set in the plan and require exactly one result envelope for each member. The result envelope should have an explicit terminal status such as Succeeded, Empty, TimedOut, or Unavailable; only Succeeded and Empty may contain a valid stage receipt.

Canonicalize typed filter predicates rather than hashing ad hoc SQL text. Bind values separately, include the authorization-policy digest, and ensure the storage query cannot omit the tenant predicate.

The evidence adapter should convert RetrievalCandidate, RankedRetrievalList, FusionReport, RerankerJudgment, DiversityReport, and CitationManifest without regenerating IDs. Validate every boundary and carry the evidence crate’s report digests into the harness receipt.

Property tests should shuffle only semantically unordered inputs. Ordered ranked hits must retain their order. Canonicalization can sort channel envelopes by enum value, but it must never sort a ranking by candidate ID.

For prompt-injection evidence, the correct outcome is not “reject every suspicious sentence.” Preserve the bytes, provenance, and locator; render it in an evidence/data block; keep it below developer and system authority; and test that provider lowering preserves that distinction.

Check your understanding

Can two candidates share a logical ID if their content digest differs?
Does higher top_k guarantee higher end-to-end answer quality?
Is a vector index snapshot sufficient without its embedding release?
Where should tenant authorization occur?
When should exact search replace approximate search?
What does RRF preserve, and what information does it discard?
Why must reranker judgments cover the complete declared pool?
Can MMR remove evidence required by a behavior contract?
Which receipt proves retrieval occurred, and which manifest proves evidence entered the prompt?
How would you distinguish ingestion recall, retrieval recall, and context inclusion in an eval?

Completion checklist

Authoritative references

Apache Lucene, BM25Similarity
pgvector, indexing, filtering, iterative scans, and recall guidance
Cormack, Clarke, and Buettcher, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods”
Carbonell and Goldstein, “The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries”
mosaic-evidence::retrieval, deterministic fusion, reranking, diversity, and citation implementation
mosaic-harness::retrieval_contract, snapshot-bound harness contract
Context Manifests and Budget Allocation
Context Engineering and Memory