Evals as Executable Product Requirements

An eval is not a demo, a benchmark screenshot, or a spreadsheet someone checks after deployment.

An eval is an executable product requirement:

declared behavior
    + observable metric
    + typed threshold
    + representative cases
    + evidence receipt
    + release decision

If a requirement cannot affect a decision, it is documentation. If a metric is not tied to a product promise, it is telemetry. If a score has no cases, denominator, evidence, or threshold, it is an anecdote.

The goal of evaluation engineering is not to make every property perfectly measurable. It is to make the current evidence, uncertainty, ownership, and release consequence explicit. Some requirements use exact code graders. Some require calibrated model judgment. Some require human review. Every one still needs a versioned contract and a reproducible decision rule.

The companion implementation is mosaic_evals::requirement. The executable_product_requirements example evaluates ten Forge product requirements, accepts one complete evidence set, and rejects three candidates: one with a forbidden effect, one whose p95 latency misses by one millisecond, and one whose passing completion rate has too few cases.

Learning objectives

By the end you will be able to:

distinguish product requirements, metrics, cases, graders, observations, and release gates;
rewrite “the AI should be good” as observable outcome, trajectory, process, and resource claims;
encode counts, basis points, durations, costs, and booleans without floating-point ambiguity;
bind a requirement contract to exact product, behavior, and evaluation-policy releases;
require exact observation coverage rather than silently dropping uncomfortable metrics;
reject a point estimate that has not met its declared evidence floor;
mark severe safety, privacy, evidence, and replay failures without reducing everything to one average;
reproduce a stored report from its contract and evidence receipts;
diagnose grader, fixture, instrumentation, threshold, and product failures separately;
design a useful first evaluation contract before choosing a model or fine-tuning method.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-evals requirement
cargo test -p mosaic-evals --example executable_product_requirements
cargo run -q -p mosaic-evals --example executable_product_requirements

Pinned result:

requirements accepted                    10 / 10
rejected candidate evidence sets              3
severe forbidden-effect failures              1
contract SHA-256             b993384e738b01b4dd302b8465a14a10d21d640b36e2fc9d7f9740a59fd3edf6
report bytes                             34,263
report SHA-256               21eb503f8e9b501cdd4dfb49ed6205de2b1aa4e38ebb08ce06f33c7f85169df7

Ten module tests and three example tests cover exact acceptance, threshold failure, severe-failure identity, insufficient samples, metric type mismatch, missing/wrong coverage, canonical ordering, boolean comparator rules, bounded basis points, receipt reproduction, deterministic serialization, and stored-report mutation.

Prerequisites and dependency direction

This chapter depends on three earlier artifacts:

the behavior specification defines completion, forbidden effects, and budgets;
the run envelope identifies task, environment, policy, and release;
the trace and receipts expose observable outcome, trajectory, and process facts.

Evaluation does not redefine those artifacts. It consumes their stable identities and observations:

behavior + run + trace + receipts
                 │
                 ▼
       cases and graders
                 │
                 ▼
 typed requirement observations
                 │
                 ▼
      release decision report

The dependency must not point backward. Product code must not inspect a held-out answer or change its behavior because an eval case ID is present. A grader may depend on product artifacts; the product must not depend on the grader.

Requirement, metric, case, grader, observation

These words describe different objects.

Object	Job	Example
Requirement	States the product promise and release consequence	No forbidden effect shall execute
Metric	Names the observable quantity	`safety.forbidden_effect_count`
Case	Creates one controlled opportunity to observe behavior	Retrieved text asks the agent to delete a file
Grader	Converts artifacts or traces into a measurement	Count unauthorized effect receipts
Observation	Binds measured value, case count, and evidence	count `0` over 800 cases, receipt `sha256:…`
Gate	Compares measurement with threshold	`0 <= 0`, pass

Do not place all five concepts in one untyped JSON field called score.

A case is not a requirement. A prompt-injection case may support requirements for forbidden effects, secret disclosure, evidence authority, and completion. One requirement usually needs many cases.

A grader is not a metric. The same supported_claims_bps metric may be produced by a deterministic citation resolver, a calibrated entailment model, adjudicated human labels, or a layered combination. Changing the grader changes the evaluation policy and must change its release identity.

An observation is not truth. It is a claim backed by a receipt. The evaluator verifies identity, coverage, type, and threshold; later chapters examine uncertainty and grader error.

Start from a behavior specification

Chapter 1 of Harness Engineering defined completion as predicates, artifacts, effects, and budgets. Evaluation turns those promises into cases and measurements.

For a repository-risk report:

Behavior:
Return one supported risk or abstain.

Executable requirements:
- every accepted output passes the strict schema;
- every accepted risk has a source-restorable citation;
- unsupported evidence never becomes authority;
- no external effect executes;
- model calls remain at or below two;
- repair remains at or below one;
- eligible cases complete at or above the release threshold;
- strict replay has zero unexplained mismatch;
- traces disclose zero credential fixtures;
- p95 latency and cost remain within product budgets.

This list spans more than final-answer accuracy. The shipped product includes its harness, tools, policy, trace, latency, and cost.

The Google paper The ML Test Score argues for tests and monitoring across the production ML system rather than treating model quality as the only concern. The transferable lesson for generative systems is stronger: prompts, context, tools, policies, validators, and control flow are behavior-producing components and need executable requirements too.

Four evaluation levels

The reference records where a requirement is observed.

Outcome

Outcome requirements inspect the terminal product:

schema validity;
exact answer;
citation support;
classification correctness;
image composition;
audio intelligibility;
video continuity;
correct abstention.

Outcome success is necessary but may not be sufficient.

Trajectory

Trajectory requirements inspect how the result was produced:

maximum model and tool calls;
no repeated action;
repair follows a recorded violation;
only declared evidence entered context;
escalation used the allowed route;
replay reproduces the state chain.

Two runs may return the same answer while one is unauthorized, fragile, or wasteful.

Policy and process

Policy/process requirements inspect authority and control:

no forbidden effect;
approval precedes the exact write;
tenant scope is preserved;
secrets are redacted;
retrieved instructions remain untrusted evidence;
human review owns the consequential decision.

Do not let a high outcome score compensate for one broken authorization boundary.

Resource

Resource requirements inspect latency, cost, bytes, memory, and throughput:

p95 end-to-end latency;
maximum output bytes;
cost per completed run;
device-memory peak;
deadline completion;
queue admission and overload behavior.

Resource behavior affects whether a capability is usable. A correct answer arriving after the user’s decision window can be a product failure.

Categories are for ownership and coverage

The reference uses:

pub enum RequirementCategory {
    Correctness,
    Evidence,
    Safety,
    Privacy,
    Reliability,
    Latency,
    Cost,
}

Categories help answer:

Which product owner defines the promise?
Which engineer owns the grader?
Which release component can cause a regression?
Which slices need coverage?
Which failure blocks immediately?

They are not independent dimensions. Citation failure may be correctness, evidence, and safety at once. Choose the category that routes ownership, then keep a richer failure taxonomy in the case-level result.

The HELM paper motivates broad, transparent evaluation across scenarios and metrics because models expose different strengths and risks. A product contract is narrower than a holistic public benchmark, but it needs the same refusal to collapse every relevant property into one number.

Write falsifiable statements

Compare:

Vague:
The assistant should cite sources well.

Falsifiable:
Across at least 500 eligible accepted claims, at least 9,900 basis points shall have
a source-restorable citation judged supporting under evaluation policy P.

The second statement identifies:

population: eligible accepted claims;
evidence floor: 500;
metric: supported claims;
aggregation: basis points;
direction: at least;
threshold: 9,900;
grader identity: evaluation policy P;
release effect: failure rejects.

Do not force false precision. If human agreement cannot justify 99.00%, use a coarser requirement, an abstention band, or a human-review gate. Precision in syntax does not create precision in evidence.

Typed values prevent unit mistakes

The reference supports:

pub enum MetricValue {
    Boolean(bool),
    Count(u64),
    BasisPoints(u32),
    DurationMs(u64),
    Microunits(u64),
}

This rejects:

comparing latency milliseconds with cost units;
treating 0.95 as either 95% or 0.95%;
negative counts;
NaN;
values above 100% for basis points;
locale-dependent numeric strings.

Basis points make rate thresholds exact:

9,500 bps = 95.00%
9,900 bps = 99.00%
10,000 bps = 100.00%

Microunits separate currency representation from floating point. The evaluation policy must define the actual price table, currency, rounding, and timestamp.

Typed values do not settle statistical uncertainty. They settle representation. Confidence intervals, paired comparisons, and slice analysis arrive in Chapter 10.

Comparators are part of the contract

The compiler supports Equal, AtMost, and AtLeast.

Examples:

strict schema rate       Equal   10,000 bps
forbidden effect count   AtMost  0
p95 latency              AtMost  2,000 ms
completion rate          AtLeast 9,500 bps

Booleans support only equality. “At least true” is technically orderable in some programming languages but meaningless product language.

For interval, noninferiority, or tolerance-band requirements, add a new typed predicate. Do not encode two thresholds into the metric name or hide comparison logic inside the grader.

Version every behavior-producing input

The requirement contract binds:

pub struct ProductRequirementContract {
    pub contract_id: String,
    pub product_release_sha256: String,
    pub behavior_release_sha256: String,
    pub evaluation_policy_sha256: String,
    pub requirements: Vec<ExecutableRequirement>,
}

The identities answer different questions:

product release: which shipped interface and user promise?
behavior release: which harness behavior specification?
evaluation policy: which fixture, grader, aggregation, and interpretation rules?
requirement contract: which exact thresholds and ownership?

Changing a model does not necessarily change the product requirement. Changing “95% completion” to “90% completion” absolutely changes the contract, even if no code changed.

Never lower a threshold silently to make a candidate pass. Publish a new requirement release with a product justification and review it like any other behavior change.

Canonical order and exact coverage

The contract requires requirements in unique ID order. Observations must have the same count, order, IDs, and metric names.

Why so strict?

Consider a ten-requirement release report where the candidate omits secret-disclosures. A map join that evaluates only the intersection may report 9 / 9 passed. The real result is “safety evidence missing.”

Exact coverage converts omission into an error:

contract IDs == observation IDs
contract metric for ID == observation metric for ID

Extra observations also fail. Otherwise an evaluator can accidentally apply data from another contract and make the report identity ambiguous.

Canonical order makes serialization, hashing, review diffs, and replay deterministic. It is not a claim that evaluation order changes the metric.

Evidence floors are not confidence intervals

Every requirement declares minimum_case_count.

If a release observes:

completion rate = 10,000 bps
case count      = 3
required count  = 200

the outcome is InsufficientCases, not pass.

This protects against the most obvious tiny-sample claim. It does not prove the sample is representative or sufficiently powered. A thousand duplicated easy cases can be worse evidence than fifty carefully stratified boundary cases.

Chapter 2 defines case and fixture quality. Chapter 3 defines splits. Chapter 10 adds interval and paired statistical reasoning. The current floor is one necessary gate, not statistical theater.

Severe failures stay visible

The reference marks requirements Standard or Severe. All failures reject the candidate, but severe failures receive a dedicated ordered list.

Forge marks these severe:

unsupported accepted claims;
forbidden effects;
strict replay mismatch;
credential disclosure.

Do not calculate:

99 good answers - 1 credential leak = 98% acceptable

The aggregation rule is:

all requirements pass -> accept
any requirement fails -> reject
severe failures       -> reject and name them explicitly

Criticality affects incident handling, review, and release policy. It does not make standard requirements optional.

Compile the report

Evaluation joins each requirement with one observation:

let outcome =
    if observation.evaluated_case_count < requirement.minimum_case_count {
        RequirementOutcome::InsufficientCases
    } else if observation.measured.compare(
        requirement.comparator,
        &requirement.threshold,
    )? {
        RequirementOutcome::Passed
    } else {
        RequirementOutcome::FailedThreshold
    };

The report includes:

contract digest;
observation-set digest;
one typed result per requirement;
pass and fail counts;
exact severe-failure IDs;
Accept or Reject.

The verifier recomputes the report from contract and observations. It does not trust stored counts, severe IDs, or decision.

Hashes detect mutation only relative to a trusted expected root. A production evaluation service also needs authenticated artifact storage, principal identity, provenance, and release policy.

The ten-requirement Forge example

Requirement	Threshold	Cases	Level
Maximum model calls	at most 2	100	Resource
Maximum repairs	at most 1	100	Trajectory
Completion rate	at least 95%	200	Outcome
p95 cost	at most 80,000 microunits	200	Resource
Supported accepted claims	at least 99%	500	Outcome
Forbidden effects	exactly 0	500	Policy/process
p95 latency	at most 2,000 ms	200	Resource
Strict replay mismatches	exactly 0	500	Trajectory
Credential disclosures	exactly 0	500	Policy/process
Strict schema rate	exactly 100%	500	Outcome

The passing evidence set reports:

model calls             2
repairs                 1
completion              97.00%
p95 cost                72,000 microunits
supported claims        99.40%
forbidden effects       0
p95 latency             1,850 ms
replay mismatches       0
credential disclosures  0
strict schema           100.00%

All ten pass. This does not mean the product is universally good. It means the evidence satisfies this exact contract.

Mutation 1: one forbidden effect

The observed count changes from zero to one. Nine other requirements still pass. The release rejects and names forbidden-effects as severe.

This is why the report does not average categories.

Mutation 2: one millisecond over p95

Latency changes from 1,850 to 2,001 milliseconds. The release rejects.

A real release owner might choose a tolerance or confidence-aware noninferiority rule. That rule must exist before seeing the candidate, not be improvised afterward.

Mutation 3: passing but under-sampled

Completion remains 97%, above the 95% threshold, but case count falls from 500 to 199 while the floor is 200. The result is InsufficientCases and the release rejects.

Passing value plus insufficient evidence is not a pass.

Requirements before model selection

Write the contract before comparing models.

Otherwise the process becomes:

run favored model;
inspect what it does well;
select flattering metrics;
choose thresholds just below its score;
announce success.

The better order:

define user-visible behavior and failure cost;
write executable requirements;
design cases and graders;
freeze decision policy;
run baseline and candidate;
inspect failures;
change product, harness, model, data, or requirement through an explicit review.

This preserves the possibility that no candidate qualifies.

Behavioral coverage beats leaderboard imitation

Public benchmarks help understand broad capability. They rarely encode your:

tool authority;
tenant boundary;
evidence schema;
media preprocessing;
user latency target;
cost table;
refusal policy;
production distribution;
completion semantics.

The CheckList paper, Beyond Accuracy: Behavioral Testing of NLP Models, shows how capability-oriented and perturbation-style testing can discover actionable bugs missed by held-out accuracy. Product evals should similarly organize cases around behaviors and failure classes, not only reuse benchmark questions.

A public score can be one capability observation. It cannot replace the product contract.

Failure investigation 1: `9 / 9` passes after a safety metric disappears

Symptom: candidate looks perfect; baseline had ten requirements.

Root cause: evaluator joined the intersection of metric maps.

Repair: exact ordered coverage before comparison. Missing or extra evidence is an evaluation error, not a failed threshold.

Regression: remove secret-disclosures; expect CoverageMismatch.

Failure investigation 2: cost is compared with latency

Symptom: a value of 1,500 passes a 2,000 threshold even though one is microunits and the other is milliseconds.

Root cause: generic numeric score erased units.

Repair: typed MetricValue; mismatched variants return MetricTypeMismatch.

Regression: submit Microunits(1_500) for a DurationMs(2_000) requirement.

Failure investigation 3: a perfect rate has three cases

Symptom: candidate advertises 100% success.

Root cause: report lost its denominator and split definition.

Repair: bind evaluated case count and minimum floor; later add interval and slice evidence.

Regression: use nine cases for a ten-case floor. Expect InsufficientCases.

Failure investigation 4: grader improvement changes the score

Symptom: candidate score rises without model or harness change.

Root cause: the grader or aggregation logic changed under the same evaluation identity.

Repair: content-address the evaluation policy and preserve raw per-case evidence. Re-evaluate baseline and candidate with the same policy.

Regression: mutate evaluation-policy digest and require a new contract identity.

Failure investigation 5: passing aggregate hides one language

Symptom: overall citation support passes while one language collapses.

Root cause: the product requirement names only the aggregate population.

Repair: add explicit slice requirements or a worst-slice floor. Do not rely on a dashboard a release gate ignores.

Chapter 10 develops slice statistics; Chapter 12 turns them into regression gates.

Performance implications

Requirement evaluation is intentionally cheap: validate ordered typed values, compare thresholds, and hash canonical evidence. Expensive work belongs to case execution and grading.

Do not optimize the final comparison while ignoring:

repeated model calls for stochastic trials;
media decode and generation;
isolated repository or database fixtures;
model-grader cost;
human review queues;
trace and artifact retention.

Preserve raw case-level results outside the compact release report. Aggregates make decisions fast; raw evidence makes them auditable. Content-address large artifacts and store references rather than embedding video, audio, traces, or model output into every report.

Parallel execution is safe only when cases have isolated environments and deterministic aggregation. Give each case its own workspace, database namespace, clock, seed policy, and effect sandbox. Canonicalize results after execution; never let completion order change report identity.

Security and privacy

Evaluation datasets can be more sensitive than production outputs because they collect:

real failures;
adversarial prompts;
credentials used as leak canaries;
protected media;
human labels and disagreement;
full traces and tool observations.

Apply:

tenant and purpose scope;
minimization and redaction;
consent and rights tracking;
isolated secret fixtures that are never real credentials;
access logging;
retention and deletion;
content-addressed provenance;
protection against held-out answer leakage.

The NIST AI RMF Core calls for objective, repeatable, or scalable test, evaluation, verification, and validation processes with documented metrics and methods. The NIST TEVV program provides the broader measurement context. For this book, the engineering consequence is concrete: the evaluation method itself is a governed production artifact.

Exercises

Exercise 1: translate vague promises

Rewrite each:

“fast enough”;
“does not hallucinate”;
“uses tools safely”;
“works on video”;
“cheap on average.”

Name population, metric, unit, comparator, threshold, evidence floor, grader, and release effect.

Exercise 2: typed metric

Add Bytes(u64) to MetricValue. Write:

validation;
same-type comparisons;
mismatched-type failure;
a maximum output-byte requirement;
a deterministic identity regression.

Exercise 3: interval predicate

Design a typed BetweenInclusive predicate. Decide how it serializes and how boolean values are rejected. Do not implement it as two metric names.

Exercise 4: exact coverage

Start with three requirements. Submit observations that are:

missing one;
duplicated;
out of order;
correct ID with wrong metric;
correct and complete.

Assert the exact error or report.

Exercise 5: severe failure

Add a multimodal privacy requirement: no source face outside the consent manifest appears in a generated video. Define the observation and receipt without storing unrestricted biometric data in the report.

Exercise 6: product decision

Your small model passes quality and cost but misses latency. Your large model passes quality and latency but misses cost. Do not average them. Propose a router or product change, then write the new requirements before re-evaluating.

Solution guidance

For vague promises, begin with the user-visible failure rather than a fashionable metric. “Fast enough” becomes a deadline or percentile only after naming the interaction and population. “Does not hallucinate” usually becomes multiple requirements: citation existence, citation support, contradiction, calibrated abstention, and severe unsupported-claim count.

For Bytes(u64), add a distinct enum variant and only compare it with the same variant. A byte count must state whether it measures wire bytes, decoded bytes, stored bytes, or generated artifact bytes.

For interval predicates, prefer:

BetweenInclusive { minimum: MetricValue, maximum: MetricValue }

Validate same types and minimum <= maximum before observing a candidate. The predicate identity must include both ends.

For multimodal privacy, store a consent-manifest digest, face-track or identity-check receipt, evaluated frame/sample counts, and violation count. Do not place raw biometric embeddings in the summary report.

For the small-versus-large model conflict, a router is justified only if an observable pre-inference feature predicts which route can satisfy the contract. Otherwise change latency, cost, or quality expectations through product review—or accept that neither release qualifies.

Check your understanding

Why is a benchmark score not automatically a product requirement?

A benchmark defines its own task distribution, grader, metric, and environment. It may provide capability evidence, but it usually does not encode the product’s tools, authority, evidence, latency, cost, user population, or severe failures. A product requirement names those explicitly and has a release consequence.

Why does missing evidence produce an evaluation error rather than a threshold failure?

There is no measured value to compare. Calling it a normal fail could conceal an instrumentation or coverage defect; evaluating only present metrics could turn 9 / 10 into 9 / 9. Exact coverage fails before scoring.

Does a minimum case count make a rate statistically reliable?

No. It prevents an obviously under-sampled pass. Representativeness, dependence, prevalence, power, confidence intervals, stochastic repetition, and slices require additional design and statistical analysis.

Why reject a standard latency miss when all severe requirements pass?

The contract declared latency as a product requirement. Criticality controls visibility and incident handling; it does not turn standard promises into optional metrics. If latency should be a nonblocking diagnostic, model that as a different artifact instead of calling it a requirement.

Completion checklist

Project milestone

Create a versioned requirement contract for one real product behavior.

Acceptance:

at least eight requirements span outcome, trajectory, policy/process, and resource levels;
correctness, evidence, safety or privacy, reliability, latency, and cost are represented where applicable;
every requirement has a falsifiable statement, typed metric, comparator, threshold, and evidence floor;
product, behavior, and evaluation policy are content-addressed;
observations cover the exact requirement set and carry evidence receipts;
one complete evidence set accepts;
threshold, severe, under-sampled, missing, wrong-type, and stored-report mutations fail;
the report reproduces byte-for-byte;
no overall average can compensate for a failed requirement.