Multimodal Retrieval, Reranking, Diversity, and Citation

A search result is not evidence merely because it is relevant.

It may be a transcript that paraphrases the audio, a thumbnail sampled seconds away from the event, a near-duplicate crop of the same diagram, or a highly similar embedding from a model release that is incompatible with the query vector. A result may cite the correct file but the wrong byte span. A video result may preserve the right presentation interval but omit the clock map needed to return it to source PTS. A model can produce a persuasive answer from all of these and still be ungrounded.

A production retrieval system must keep four questions separate:

Candidate generation: which bounded evidence units might matter?
Reranking: which candidates best answer this query under a stronger model?
Diversity: which small set adds distinct useful evidence instead of repeating one source?
Citation: can every selected item be restored to exact acquired evidence with its lineage and coordinate maps?

This chapter implements that funnel without pretending that BM25, cosine similarity, audio-text alignment, and a cross-encoder produce comparable numbers. Independent retrievers contribute ordered lists. Weighted reciprocal rank fusion combines their ranks. A versioned reranker must judge every fused item exactly once. A bounded maximal marginal relevance pass trades relevance against redundancy inside one declared similarity space while enforcing per-evidence and per-modality caps. Finally, a citation manifest binds each chosen candidate to artifact digest, typed locator, lineage digest, named clock, clock-map digest, and geometry-map digest.

The companion does not generate embeddings or run an approximate-nearest-neighbor index. Those runtime integrations begin in the next chapters. Its boundary starts at validated retriever outputs, where orchestration bugs are both common and independently testable.

Learning objectives

By the end you will be able to:

choose retrieval units for text, images, audio, and video without erasing source coordinates;
distinguish an acquired modality from a derived textual proxy such as OCR or an ASR transcript;
distinguish same-modality embedding spaces from genuinely shared cross-modal spaces;
explain why raw BM25, cosine, distance, and cross-encoder scores cannot be averaged casually;
normalize score direction while retaining the original retriever’s score-space identity;
implement deterministic weighted reciprocal rank fusion using rank rather than score magnitude;
budget first-stage depth against approximate-index recall and reranker cost;
require complete, unique reranker judgments for a frozen candidate set;
distinguish reranker relevance from factual support and citation correctness;
derive maximal marginal relevance and explain the meaning of its trade-off parameter;
combine semantic redundancy with explicit source and modality caps;
explain why “one result per modality” is not a substitute for semantic diversity;
construct source-restorable citations for byte, time, image, and video-region locators;
carry temporal clock/map identity and spatial geometry-map identity with results;
bind each stage to an immutable report identity;
measure retrieval recall, ranking quality, diversity, citation precision/recall, and latency separately;
diagnose missing evidence without immediately changing the answer model;
defend the funnel against tenant leakage, retrieval poisoning, prompt injection, and citation spoofing.

Prerequisites and dependency order

Read all previous Part 4 chapters:

Evidence envelopes provide immutable artifact identity.
Provenance graphs distinguish source records from OCR, transcript, crop, and other derivatives.
Text normalization gives text chunks exact normalized and original byte coverage.
Image processing defines oriented regions and embedding contracts.
Audio processing preserves sample clocks and ranged speech/events.
Video processing preserves presentation intervals and sampled-frame reasons.
Timeline and geometry alignment maps native clocks into canonical time and model boxes back into source pixels.

Also review task output contracts and provider-neutral capabilities. A retriever, embedder, and reranker have different input/output contracts even when one provider serves all three.

Retrieval precedes the harness chapters because retrieved evidence becomes harness context. It precedes evaluation because a generation failure cannot be attributed responsibly until candidate-generation, ranking, selection, and citation outcomes can be replayed independently.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evidence --example multimodal_retrieval

The query is:

explain-structured-cancellation-with-evidence

The fixture contains five candidates:

Candidate	Modality	Source	Locator
`text-primary`	text	structured-concurrency guide	bytes `[1200,1520)`
`text-near-duplicate`	text	same guide	overlapping bytes `[1420,1760)`
`audio-explanation`	audio	meeting recording	program `[42000,51500)` ms
`video-demo`	video	cancellation demo	program `[43000,50000)` ms plus source region
`image-diagram`	image	task-tree diagram	oriented pixel region

Three retrievers return all five in different orders:

lexical BM25;
transcript-oriented text semantic retrieval;
a shared vision-language retriever.

Weighted reciprocal rank fusion produces:

text-primary
video-demo
audio-explanation
text-near-duplicate
image-diagram

The stronger reranker produces:

text-primary
audio-explanation
video-demo
image-diagram
text-near-duplicate

The top two textual candidates overlap and refer to the same evidence artifact. A diversity pass with relevance weight 0.65, maximum one candidate per evidence record, and maximum two per modality selects:

1. text-primary
2. image-diagram
3. video-demo
4. audio-explanation

This is not a hard-coded “one of each modality” policy. The image is selected because its relevance/redundancy trade-off beats the alternatives; the duplicate is ineligible because the same evidence record is already represented. The final citation manifest happens to contain all four modalities and records every stage digest.

First principles: retrieve evidence units, not loose content

Candidate identity

A candidate is the smallest independently rankable evidence unit. Its ID is not the artifact digest. One artifact can yield many candidates:

text blocks, headings, paragraphs, tables, or code units;
image regions or whole images;
speech turns, acoustic events, or fixed audio windows;
shots, sampled frames, tracked regions, or aligned audiovisual windows.

The candidate record in the companion includes:

pub struct RetrievalCandidate {
    pub candidate_id: String,
    pub evidence_id: EvidenceId,
    pub modality: EvidenceModality,
    pub artifact_sha256: Sha256Digest,
    pub locator: Locator,
    pub lineage_sha256: Sha256Digest,
    pub clock_id: Option<String>,
    pub clock_map_sha256: Option<Sha256Digest>,
    pub geometry_map_sha256: Option<Sha256Digest>,
}

candidate_id identifies an indexed unit. evidence_id identifies its evidence graph node. artifact_sha256 identifies bytes. locator addresses coverage. lineage_sha256 identifies how a derived candidate came to exist. Temporal candidates additionally require a named clock and clock map. A geometry map is retained when a spatial locator passed through model space.

The validator rejects a time locator on image evidence, a video region on text, or a temporal locator without clock identity. “42 seconds” is not source-restorable when the clock is unnamed.

Chunking is a retrieval model

Chunking changes what can be found. It is not neutral preprocessing.

A very large text chunk may contain the answer but dilute its embedding and overflow the reranker’s input. A very small chunk may match keywords but omit the condition that reverses their meaning. A sampled video frame may show the result but miss the preceding action. A fixed audio window may split a speaker turn. A crop may remove the legend needed to interpret a plot.

Record:

chunker release and parameters;
structural boundaries used;
overlap and parent relationship;
source locator;
token count under the actual embedder/reranker tokenizer;
modality-specific context such as neighboring frames or transcript;
derivation fingerprint.

Evaluate chunking as a first-stage recall decision. If gold evidence cannot exist as one retrievable unit, no reranker can recover it.

Derived text does not change modality

An audio candidate may be indexed through its ASR transcript. A video candidate may be indexed through captions and OCR. Those are retrieval views, not proof that the source is text.

Keep:

source modality: audio
retrieval view: ASR text
retrieval view lineage: audio → decoder → resampler → ASR
selected citation: audio source range + transcript derivative

This lets an answer show the transcript for accessibility while citing the exact recording interval. Treating every ASR segment as a text artifact loses channel, timing, speaker, acoustic event, and model-error context.

Multi-vector candidates

One candidate can have several retrieval views:

original text embedding;
OCR embedding;
visual embedding;
transcript embedding;
audio embedding;
metadata keywords;
sparse lexical terms;
late-interaction token vectors.

Do not concatenate them merely because a vector database accepts one array. Their dimensions, training objectives, normalization, and meanings differ. Store a contract per view:

model release
processor/tokenizer release
embedding space ID
dimension
normalization
distance function
input derivation
index release

The CLIP paper demonstrates a learned shared image/text representation through contrastive image-caption training, enabling text-to-image comparison. It does not imply that arbitrary text and image embeddings from unrelated encoders share a space. See Learning Transferable Visual Models From Natural Language Supervision.

Candidate generation

A high-recall first stage should exploit complementary failure modes.

Lexical retrieval

Sparse term retrieval is strong for:

exact identifiers and error codes;
uncommon API names;
quoted phrases;
numbers and version strings;
names absent from an embedder’s training;
queries whose precise token matters.

Tokenization, stemming, stop-word policy, field boosts, length normalization, and language analyzer are part of the release. OCR/ASR confidence can influence indexing, but do not delete uncertain tokens without retaining the original observation.

Dense semantic retrieval

Dense retrieval can match paraphrase and concept similarity. It depends on:

query/document encoder relationship;
pooling and normalization;
truncation;
domain match;
index distance configuration;
approximate search parameters;
vector and index release identity.

Dense Passage Retrieval is a canonical dual-encoder design in which questions and passages are encoded independently for efficient maximum-inner-product search: Dense Passage Retrieval.

One vector per passage compresses token-level detail. Late-interaction systems retain multiple token vectors and score query/document interactions more richly at higher storage and compute cost. ColBERTv2 discusses this quality/space trade: ColBERTv2.

A shared space may support:

text query → image or video frame;
image query → text or image;
audio query → text/audio, if trained for that relationship;
video query → temporally pooled video/text.

“Multimodal model” is not a sufficient contract. Test each direction. An image-text model may represent objects well and temporal actions poorly. An audio-text model trained on environmental sounds may not retrieve speaker meaning. A frame encoder does not automatically represent a five-second event.

For video, index multiple levels:

frame or shot visual view;
OCR/caption view;
aligned transcript view;
motion/action view;
parent clip summary.

Return one candidate ID with multiple contributing retrieval views when they refer to the same evidence unit. Otherwise diversity sees artificial duplicates.

Metadata and structural retrieval

Filters and boosts can use:

tenant and authorization scope;
source collection;
time range;
language;
modality;
speaker or camera, when verified;
document section;
trusted-source policy;
usage rights;
model-supported input constraints.

Authorization is a filter, not a post-ranking cleanup. Query only permitted partitions or apply security predicates before candidate material leaves the storage boundary. Removing unauthorized hits after ANN search can leak counts, timing, labels, or content into reranking.

Temporal and spatial expansion

A precise hit often needs neighbors:

preceding/following text block;
adjacent speech turn;
full video shot around a sampled frame;
parent image around an OCR box;
aligned audio for a video region.

Expansion happens after the seed hit and under its own budget. It creates context candidates with explicit relationships; it must not overwrite the seed rank. Test whether expansion improves answer support rather than assuming more context is better.

Approximate nearest-neighbor search is a recall budget

ANN indexes trade exactness for latency/memory. HNSW constructs a navigable multi-layer proximity graph; search and construction parameters affect recall and cost. See the original HNSW paper.

Operational evaluation needs:

exact-search reference on a representative bounded subset;
recall@candidate-depth by query and modality slice;
latency and memory at each search parameter;
index build and update behavior;
deletion/tombstone behavior;
filter selectivity;
deterministic tie policy;
index/model/processor compatibility checks.

If the target result is absent from the first-stage pool, raising reranker quality cannot help. Tune candidate depth and ANN effort jointly with reranker cost.

Do not report only end-to-end answer accuracy. A failing query should reveal whether:

gold candidate absent from index
→ absent from ANN results
→ present but fused below rerank depth
→ reranker misordered
→ diversity removed
→ context pack truncated
→ answer ignored evidence
→ citation attached incorrectly

Score spaces are not exchange rates

These values are not naturally comparable:

BM25 = 12.7
cosine = 0.81
L2 distance = 0.43
cross-encoder logit = 4.2
provider relevance = 0.96

Even two cosine scores can differ across models, query distributions, or normalization. L2 has the opposite direction from similarity. A softmax probability over one candidate batch changes when the batch changes.

Calibrated score fusion is possible, but it requires held-out data and a stable distribution. Simple min-max normalization per query is unstable: adding one outlier changes every normalized score. Z-scores assume a meaningful distribution and can fail on short lists. Learned fusion can work but becomes another versioned model with training/evaluation obligations.

The companion retains each retriever’s score_space, validates a higher-is-better fixed-point diagnostic score, and fuses rank.

Weighted reciprocal rank fusion

For candidate d, ranked lists R, list weight wᵣ, constant k, and one-based rank rᵣ(d):

RRF(d) = Σᵣ wᵣ / (k + rᵣ(d))

Candidates missing from a list contribute zero. A smaller k makes the very top ranks more dominant. Larger k makes differences among early ranks gentler. Weights express a reviewed policy; they are not learned truth.

RRF is attractive when independent systems have incompatible raw scores. Cormack, Clarke, and Buettcher introduced and evaluated the method in Reciprocal Rank Fusion.

The companion uses checked integer arithmetic:

contribution =
    weight_basis_points × 1,000,000
    / (reciprocal_rank_constant + rank)

This produces deterministic sortable units. It is a ranking score, not a probability.

Every ranked list must:

have a unique list ID;
name retriever release and score space;
contain known, unique candidates;
have descending diagnostic scores;
stay within list and candidate bounds;
use a nonzero bounded weight.

The full list inputs receive a stable digest. The fusion report records every retained candidate’s contributing list, rank, and contribution. Changing a retriever release, raw diagnostic score, ordering, weight, or k changes report identity.

Candidate depth is part of the experiment

The fixture fuses five candidates. Production systems may retrieve hundreds and rerank tens. Record:

retrieval depth per list
fusion output depth
rerank depth
diversity pool depth
final context depth/bytes/tokens/media duration

Truncation is legitimate only when visible. FusionReport records both unique and omitted candidate counts. A result beyond the output limit did not “lose”; it was outside the declared downstream budget.

Reranking

First-stage encoders optimize reusable search. A reranker can jointly inspect query and candidate, often producing better ordering at higher per-pair cost.

Possible rerankers include:

text cross-encoder over query and passage;
vision-language cross-attention over query and image;
audio-text or video-language model;
rule-based exact-identifier boost;
ensemble combining modality specialists;
LLM/VLM judge with a strict score schema.

Freeze the batch

The companion requires exactly one RerankerJudgment for every fused candidate:

pub struct RerankerJudgment {
    pub candidate_id: String,
    pub relevance_micros: i32,
}

Missing, duplicate, extra, or out-of-range judgments fail. Without this rule, a timeout can silently remove the hardest item, a duplicated response can overwrite another candidate, or a provider can reorder inputs while the caller attaches scores positionally.

Join by candidate ID. Preserve original fused rank and score. Sort by:

reranker relevance descending;
fused score descending;
candidate ID ascending.

The final tie rule is not semantically magical; it makes replay deterministic.

Relevance is not support

A reranker score answers something like “how useful is this candidate for the query?” It does not establish:

that a quoted statement is true;
that the evidence entails an answer claim;
that the source is trustworthy;
that the selected time/region is correct;
that the candidate may be shown to this user;
that the content is safe to execute as an instruction.

Use separate graders or policies for source trust, claim support, and authorization. Do not name a field confidence when its actual meaning is an uncalibrated relevance logit.

Input construction is part of the release

A multimodal reranker release must include:

model and processor;
query template;
candidate serialization;
image resize/crop and frame sampling;
audio preprocessing and maximum duration;
transcript/OCR inclusion;
truncation direction;
batch shape;
output interpretation;
quantization/runtime;
score normalization.

Changing how a video candidate is represented—from middle frame to four sampled frames—is a behavioral change even if weights are identical.

Diversity and maximal marginal relevance

Top relevance results often repeat:

overlapping chunks from one document;
adjacent frames in one shot;
the same image at several resolutions;
transcript and summary derived from one utterance;
mirrored or recompressed media;
several sources repeating one press release.

Giving an answer model five versions of the same fact wastes context and creates the appearance of corroboration.

For candidate d, query relevance Rel(d), selected set S, pair similarity Sim(d,s), and trade-off λ:

MMR(d) =
    λ × Rel(d)
    - (1 - λ) × maxₛ∈S Sim(d,s)

The first item has redundancy zero. At every next step, choose the eligible candidate with maximum MMR. Carbonell and Goldstein introduced this diversity-based reranking approach in The use of MMR.

The companion represents λ in basis points and relevance/similarity in millionths:

mmr_units =
    relevance_weight_bp × relevance_micros
    - (10,000 - relevance_weight_bp) × maximum_redundancy_micros

The number is not a probability. Pair similarity must be complete for every unordered pair in the bounded diversity pool and must name one shared similarity_space. The complete canonical matrix gets its own digest.

Why the matrix must have a contract

Do not calculate redundancy by taking cosine between:

one text-encoder vector and one image-encoder vector;
embeddings with different dimensions or releases;
normalized and unnormalized vectors;
audio-event and transcript-semantic spaces;
product-quantized stored codes interpreted as original vectors.

Use a shared cross-modal space, modality-appropriate duplicate detector, or an ensemble that produces one versioned pair score. Near-duplicate hashes, temporal overlap, source identity, and derivation ancestry can be stronger redundancy signals than semantic embeddings.

The fixture uses a declared joint-evidence-diversity-v2 space and gives the two text chunks similarity 0.99.

Caps are policy, not semantic similarity

The selection also enforces:

max_per_evidence = 1;
max_per_modality = 2.

The first prevents overlapping candidates from one evidence record dominating. The second prevents one modality from exhausting the result budget. Caps are explainable product policies, not a replacement for MMR.

Hard “one per modality” selection can lower quality when a query legitimately requires several text sources and no image. Conversely, MMR alone can preserve four visually different images that all repeat the same unsupported claim. Measure both semantic and source diversity.

Diversity can remove gold evidence

Evaluate diversity separately. Record each candidate’s:

pre-diversity rank;
relevance score;
maximum redundancy and which selected item caused it;
cap eligibility;
MMR score;
selection/exclusion reason.

The current companion stores maximum redundancy but not the argmax candidate or every exclusion reason. That is a deliberate version-1 limitation. A production explainability extension should add them, include them in the digest, bump DIVERSITY_RELEASE, and test cap-blocked versus redundancy-lowered outcomes.

Citation is a data product

A citation should answer:

Which acquired bytes?
Which exact part?
Which derived path?
Which temporal/spatial coordinate systems?
Which maps restore the source?
Which retrieval/rerank/diversity run selected it?

The final CitationRecord contains:

final selection rank;
candidate and evidence IDs;
source modality;
artifact SHA-256;
typed locator;
lineage SHA-256;
optional clock ID and required map for temporal locators;
optional geometry map;
reranker relevance.

The manifest binds fusion, rerank, and diversity report digests. It rejects reports from different runs. This prevents a caller from pairing yesterday’s selected IDs with today’s ranking metadata.

A citation locator is not display text

For text, the locator identifies source byte coverage. The UI may show a normalized excerpt, but it must be derived through the text mapping.

For audio, show a player range in the named canonical clock and retain the map back to source sample frames.

For video, cite the presentation interval plus source pixel region. A thumbnail is a preview, not the citation itself.

For images, cite oriented source pixels and, when the observation came from model tensor space, retain the geometry map used for back-projection.

Do not put signed CDN URLs into stable citation identity. Store artifact identity and locator; mint access-controlled delivery URLs at presentation time.

Retrieval citation versus answer citation

The retrieval manifest says which evidence was selected. An answer needs a second mapping:

answer claim/span → one or more citation record IDs

The answer generator should not attach every retrieved source to the final paragraph. For each claim, verify:

citation entails or directly supports the claim;
cited range contains the support;
source trust is sufficient for the claim type;
contradictory evidence is represented;
citation was actually available to the generator;
no unsupported detail was added.

Retrieval provenance and claim attribution are related, not identical.

Stable identity through the funnel

The companion uses domain-separated SHA-256 reports:

ranked lists
  → ranked_lists_sha256
  → weighted RRF report_sha256
  → complete rerank report_sha256
  → complete similarity_matrix_sha256
  → diversity report_sha256
  → citation manifest_sha256

Each stage includes its upstream identity. Stable hashing uses explicit field order, length-prefixed strings, fixed-endian integers, and enum tags. It does not hash debug output or map iteration order.

Report identity is not run identity. Two executions with identical inputs and releases should produce the same deterministic report hash. Operational run ID, start time, trace ID, host, and latency belong in an execution receipt that references the report.

Model calls may be nondeterministic. In that case the exact returned judgments become input to the deterministic downstream report; the execution receipt must name provider request ID, resolved model release, parameters, and response digest.

Retrieval evaluation

Build judgments at evidence-unit granularity

A useful dataset contains:

query and intent;
acceptable evidence candidate IDs or source locators;
graded relevance;
required modalities, if the task truly requires them;
hard negatives;
near-duplicate groups;
contradictory sources;
trust/rights constraints;
answer claims and supporting locator pairs.

Do not label only document IDs when the product cites passages or time ranges. Document-level gold can mark a retrieval successful even when the selected chunk omits the answer.

First-stage metrics

Measure:

recall@K;
success@K;
modality/source recall;
exact-identifier recall;
temporal and spatial locator coverage;
ANN recall versus exact search;
candidate count and bytes;
p50/p95/p99 latency;
index freshness.

Recall is primary at this stage. Precision can be low because the reranker exists, but unbounded candidate pools are not free.

Ranking metrics

Use:

mean reciprocal rank when one early relevant result matters;
nDCG for graded relevance and multiple results;
precision@K;
average precision;
pairwise accuracy for rerankers;
calibration only if scores have a declared probabilistic meaning.

Report per-modality, per-language, per-source-type, query-length, OCR/ASR quality, and seen/unseen domain slices. The BEIR benchmark was designed to evaluate heterogeneous zero-shot information retrieval across datasets and tasks; its central lesson is that in-domain wins do not guarantee robust transfer. See BEIR.

Diversity metrics

Measure:

unique evidence records at K;
unique source authorities at K;
duplicate-cluster coverage;
average pairwise similarity;
subtopic or intent coverage;
gold evidence retained after diversity;
context bytes/tokens saved;
downstream answer change.

More modalities is not automatically more diversity. Four modalities derived from one false source are one lineage.

Citation metrics

Separate:

citation correctness: cited evidence supports the associated claim;
citation completeness: supported factual claims have citations;
locator correctness: cited range/region actually contains support;
citation precision: citations are not irrelevant decorations;
source quality: evidence meets trust policy;
restorability: artifact and maps reproduce the cited view.

For temporal/spatial citations, use interval overlap and region IoU only as diagnostics. A large overlap can still miss the decisive moment; a small precise region can be correct.

Counterfactual evaluation

Run ablations:

lexical only;
dense only;
cross-modal only;
no fusion weighting;
no reranker;
no diversity;
no neighbor expansion;
transcript removed;
image pixels removed;
map identities withheld.

This reveals which component supplies quality and which only adds cost.

Failure investigation: the answer misses the diagram

Suppose the source diagram directly explains task cancellation, but the answer cites only a transcript paraphrase.

Trace the funnel:

Candidate formation: Did the diagram exist as a whole-image or region candidate? Was its caption/OCR derived with valid lineage?
Embedding contract: Was the query encoded in the same shared space as the diagram?
Index membership: Did the correct vector and candidate ID enter the expected tenant/index release?
ANN recall: Does exact search find it? Does increasing search effort recover it?
Retriever rank: Was it outside the vision-language depth?
Fusion: Did list weights or k push it below rerank depth?
Reranking input: Did resize/truncation remove the relevant diagram region or legend?
Reranker output: Was the image judged once and joined by ID?
Diversity: Was it considered redundant with text under an invalid cross-space comparison?
Caps: Did a modality or source cap exclude it?
Context pack: Did media/token budgeting remove it after selection?
Generation: Was it present but ignored?
Citation mapping: Did the answer use it but fail to attach the record?

Keep the first failing boundary. Do not “fix” missing image recall by forcing one image into every answer unless evaluation supports that policy.

Security and trust boundaries

Tenant and authorization isolation

Candidate IDs, vector indexes, reranker batches, caches, and citation delivery must remain tenant-scoped. Validate authorization before retrieval. Include policy scope in cache keys. Never let a shared reranker log raw private candidates.

Retrieval poisoning

An attacker can repeat target keywords, generate visually similar decoys, manipulate OCR, or create many overlapping chunks to dominate top K. Mitigations include:

per-evidence/source caps;
duplicate clustering;
source trust features kept separate from relevance;
ingestion rate and size limits;
anomaly detection;
signed acquisition/lineage where needed;
adversarial evaluation fixtures.

Do not lower relevance score merely because a source is untrusted and then call the resulting number “semantic relevance.” Filter or expose separate policy decisions.

Indirect prompt injection

Retrieved text, OCR, transcripts, images, and audio can contain instructions aimed at the agent: “ignore prior rules,” fake tool results, or requests to exfiltrate data. Evidence content is data, not authority.

The harness must:

label untrusted context;
keep system/tool instructions outside evidence fields;
authorize every tool call independently;
escape or structure candidate serialization;
limit retrieved bytes/media;
prevent evidence from choosing tools or recipients;
log which candidate influenced a proposed action;
grade prompt-injection resistance.

An image can carry adversarial text; an audio clip can speak it; a video can reveal it in one frame. Multimodal retrieval expands the injection surface.

Citation spoofing

Never trust a model-generated candidate ID or URL. The answer may reference only citation IDs issued by the retrieval/citation service. Resolve the ID to a server-side immutable record and recheck authorization when rendering.

Resource exhaustion

Bound:

candidates and lists;
hits per list;
identifier lengths;
fusion output;
rerank batch size and bytes;
image/audio/video preprocessing;
diversity pool (O(n²) pair matrix);
final citations;
hash input lengths;
timeouts and concurrency.

The companion caps diversity at 128 candidates because a complete matrix then has 8,128 unordered pairs. Production systems can use approximate diversity or cluster summaries for larger pools, but that changes the algorithm and needs its own release/evaluation.

Performance model

Break latency into:

query preprocessing
+ parallel sparse/dense/cross-modal retrieval
+ fusion
+ candidate materialization
+ reranker preprocessing/inference
+ pair similarity or duplicate lookup
+ diversity selection
+ context/citation assembly

Fusion is approximately linear in total returned hits. Complete reranking cost grows with candidate count and media size. The companion’s MMR implementation repeatedly scans remaining candidates and looks up selected-pair similarity, which is acceptable for its bounded pool but quadratic in the worst case.

Optimize the expensive stage:

run independent retrievers concurrently under a shared deadline;
reserve separate timeouts so one modality cannot consume the whole budget;
batch reranker pairs by compatible shape;
cache immutable candidate representations by artifact/preprocessor/model identity;
precompute duplicate clusters or shared-space vectors;
short-circuit no-result lists explicitly, not as success;
preserve partial-outage state in the report;
keep final deterministic arithmetic on CPU.

The current companion rejects an empty ranked list. A production partial-outage schema should represent each retriever as succeeded, timed_out, failed, or disabled, with an error class and attempted release. Silently omitting a failed list makes a degraded run look like an intended configuration.

Test strategy

The companion includes tests that:

fuse incompatible score spaces through rank order;
reject ascending scores, duplicate candidates, and unknown IDs;
require complete unique reranker judgments;
apply deterministic reranker ties;
require a complete canonical similarity matrix;
remove a near-duplicate from the same evidence record;
validate modality-compatible locators;
preserve temporal/spatial source-restoration fields;
reject mixed report identities;
bound scores and policies;
change identities when ranking or relevance changes;
run the complete four-modality example.

Add:

Property tests

permuting input list order leaves fusion output unchanged;
changing raw scores without changing order leaves RRF ranking unchanged but changes input/report identity;
every fused score equals the checked sum of recorded contributions;
every reranked candidate appears exactly once;
every selected candidate was in the rerank pool;
per-evidence and per-modality counts never exceed policy;
citation rank is contiguous from one;
rebuilding identical input produces identical hashes.

Metamorphic tests

adding a list containing no retained candidates cannot change retained contributions, though it changes run identity;
adding the same constant to valid retriever diagnostic scores cannot change rank fusion;
setting MMR relevance weight to 10,000 makes semantic redundancy irrelevant while caps remain;
setting it to zero prioritizes minimum redundancy after the deterministic first choice;
replacing one candidate ID with a new ID changes all downstream identities;
changing a clock or geometry map changes citation identity even if rank is unchanged.

Adversarial fixtures

thousands of chunks from one artifact;
OCR keyword stuffing;
ASR hallucinated exact identifier;
image and text embeddings from incompatible releases;
reversed-distance ordering;
duplicated reranker response IDs;
NaN/provider overflow converted at the boundary;
cyclic derived lineage;
temporal locator without a clock map;
model-generated fake citation ID;
prompt injection in all four modalities.

Exercises

Exercise 1: calculate weighted RRF

With k = 60, equal weight, and two lists:

List A: X rank 1, Y rank 2
List B: Y rank 1, Z rank 2

Calculate each score without fixed-point scaling.
Rank X, Y, and Z.
Explain why the retriever raw scores are unnecessary.

Exercise 2: reason about score fusion

A BM25 retriever returns score 14.2; an image-text retriever returns cosine 0.31.

Why is 14.2 + 0.31 meaningless?
Name two defensible fusion strategies.
What evaluation data would learned/calibrated fusion require?

Exercise 3: apply MMR

Candidates A, B, and C have relevance 0.9, 0.86, and 0.80. After A is selected, similarities to A are B=0.95, C=0.20. With λ=0.6, calculate B and C MMR scores and choose the next result.

Exercise 4: design a video citation

A model selects an object box from a 224 × 224 tensor at program [2.0,2.5) seconds. List every identity and locator needed to show the exact original video evidence.

Exercise 5: diagnose low recall

Exact vector search finds gold evidence at rank 4. ANN returns it at rank 150. Fusion keeps only 50 items from that retriever; the reranker sees 40.

Which stage first loses the gold item?
Which measurements isolate the cause?
Which changes are plausible?
Why is fine-tuning the answer model irrelevant?

Exercise 6: evaluate citations

An answer has five factual claims. Four have citations. Three cited locators support their claims; one citation points to the right document but wrong paragraph. One unsupported claim has no citation.

Calculate simple claim-level citation completeness and cited-claim correctness. State what these two metrics still omit.

Solutions

Solution 1

X = 1/(60+1)
Y = 1/(60+2) + 1/(60+1)
Z = 1/(60+2)

So Y > X > Z. RRF needs only membership and one-based rank. Raw score still belongs in diagnostic evidence and release identity, but it does not enter this formula.

Solution 2

The numbers have different objectives, scales, distributions, and possibly directions. Defensible strategies include reciprocal rank fusion and a calibrated/learned fusion model trained on held-out graded relevance. Learned fusion needs representative queries, candidate judgments, stable feature definitions, leakage-safe splits, modality/domain slices, and evaluation against strong individual retrievers.

Solution 3

MMR(B) = 0.6×0.86 - 0.4×0.95 = 0.136
MMR(C) = 0.6×0.80 - 0.4×0.20 = 0.400

C is selected despite lower raw relevance because it adds substantially different evidence.

Solution 4

Retain acquired video evidence/artifact digest, track and decoded-frame identity, source PTS and duration, canonical clock ID, piecewise clock-map digest, source oriented pixel space and dimensions, tensor geometry-map digest, model-space box, conservative source back-projected region, decoder/preprocessor/model releases, lineage digest, candidate ID, and retrieval/rerank/diversity report identities. The UI mints an authorized media URL separately.

Solution 5

ANN is the first failing boundary: exact rank 4 becomes approximate rank 150, outside retrieval depth 50. Compare exact versus ANN recall across search parameters, filters, index release, and query slices. Increase ANN effort/depth, repair index/metric mismatch, partition differently, or improve embeddings after evidence. The answer model never receives the item, so tuning it cannot recover the missing source.

Solution 6

Simple completeness is 4/5 = 80% because four of five claims have citations. Correctness among cited claims is 3/4 = 75%. These omit degree of entailment, source trust, contradiction handling, locator precision, whether the generator actually saw the evidence, and severity/importance of each claim.

Completion checklist

You are done when you can:

Further deductions

First, retrieval quality is constrained by evidence engineering. Better embeddings cannot restore coordinates discarded during ingestion or retrieve a concept split across unusable units.

Second, multimodal retrieval is usually multi-view retrieval. The system gains robustness by combining lexical text, transcript semantics, visual semantics, acoustic events, and structure without pretending they are one datatype.

Third, ranking and citation have different correctness conditions. A candidate can be highly relevant yet uncitable, or precisely citable yet irrelevant. Enforce both.

Fourth, diversity must follow lineage. Different crops, transcripts, summaries, and modalities can all descend from one source. Counting them as independent corroboration is a provenance error.

Fifth, every optimization changes an evaluation surface. ANN depth affects recall, fusion affects exposure, reranker truncation affects evidence, MMR affects coverage, and context packing affects generation. Preserve stage outputs so end-to-end metrics can be decomposed.

Finally, a citation manifest is the bridge between retrieval and an AI harness. The harness can budget, serialize, quote, render, and attribute evidence only when selected content arrives with stable identity and reversible coordinates. That bridge will become the context-pack contract in later chapters.

References

Cormack, Clarke, and Buettcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.
Carbonell and Goldstein, The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.
Radford et al., Learning Transferable Visual Models From Natural Language Supervision.
Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering.
Santhanam et al., ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.
Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.
W3C, PROV-O: The PROV Ontology.

The executable source of truth is rust/rust-ai-engineering/crates/mosaic-evidence/src/retrieval.rs with the multimodal_retrieval.rs example.