Datasets and Graders

An eval is an executable product requirement.

Without one, harness changes are anecdotes. With one, you can compare prompts, tools, models, routing, and fine-tuning on the same behavior.

An evaluation case

struct EvalCase {
    id: CaseId,
    task: Task,
    environment: EnvironmentFixture,
    expected: Expectations,
    tags: BTreeSet<Tag>,
    provenance: CaseProvenance,
}

The environment matters. A coding agent needs a repository state. Signal Room needs the same logs, screenshots, audio, and traces each run. A tool-using eval that silently reads changing production data is not reproducible.

Build the dataset from behavior classes

Include:

• ordinary successful cases;
• boundary sizes and formats;
• missing or contradictory evidence;
• ambiguous requests;
• provider and tool failures;
• adversarial instructions inside retrieved content;
• cases that require refusal or human escalation;
• expensive cases that test the budget;
• previously observed production failures.

Track slices. An overall score can hide a collapse on audio, long videos, one language, or denied tool calls.

Prefer deterministic graders

Use code when the property is observable:

• JSON parses and validates;
• cited evidence exists;
• generated code compiles and tests pass;
• a database contains the expected state;
• a video cut list is in bounds and non-overlapping;
• the run stayed within budget;
• no forbidden tool was called.

Deterministic graders are fast, cheap, and explainable.

Use model graders for semantic judgments

Some properties require judgment: completeness, visual coherence, usefulness, or whether a summary preserves a subtle claim.

A model grader needs:

• a narrow rubric;
• reference evidence;
• structured output;
• examples near decision boundaries;
• calibration against human labels;
• periodic drift checks;
• an abstain or review outcome.

Do not ask “Is this good from 1–10?” Ask separate questions with anchors.

{
  "criterion": "claim_support",
  "score": 1,
  "allowed_scores": {
    "0": "contradicted or no relevant evidence",
    "1": "partially supported; important inference unstated",
    "2": "directly supported by cited evidence"
  }
}

Human review is part of the stack

Humans should review:

• disputed grader cases;
• high-risk actions;
• novel failure clusters;
• samples used to calibrate model graders;
• subjective generated media;
• candidates for fine-tuning data.

Record disagreement. It can reveal an underspecified product requirement rather than a bad reviewer.

Metrics need denominators

Report:

• pass rate with case count and confidence interval;
• per-slice results;
• severe failure count;
• cost and latency distribution;
• tool-call and retry distribution;
• grader agreement;
• abstention and escalation rate.

An average score of 0.91 across ten easy cases is not comparable to 0.89 across a thousand adversarial cases.

Avoid evaluation leakage

Keep a development set for iteration and a held-out set for decisions. If instructions, examples, or fine-tuning data contain held-out answers, the score no longer measures generalization.

Version:

• case content and environment;
• harness;
• model and parameters;
• graders;
• scoring logic.

The evaluation harness

Anthropic’s agent eval guide emphasizes the full end-to-end harness: task, environment, agent loop, transcript, and graders. That is the unit you ship and the unit you must evaluate.

Practice

Create incident-v1 with 20 cases:

• 8 routine;
• 4 missing-evidence;
• 3 contradictory-evidence;
• 3 prompt-injection;
• 2 budget-exhaustion.

Implement four deterministic graders and one calibrated model grader. Publish the slice table, not only the overall score.