Capability, Knowledge, Instructions, and Calibration

A model answers a Rust question correctly but ignores the requested JSON schema. Another model does not know today’s package release, calls an approved tool, and returns a verified answer. A third model confidently invents a private deployment fact that no prompt, model parameter, or tool could contain.

Which model is “better”?

One accuracy number cannot answer. The first has task knowledge but failed the output contract. The second correctly recognized a knowledge boundary and used the system. The third failed both epistemically and behaviorally. Treating all three as ordinary question-answering cases destroys the information needed to improve the harness.

This chapter builds a typed behavior scorer in mosaic-evals. It keeps five questions separate:

Could the system perform the task?
Was the necessary information available, and where?
Did the system take the required action?
Did it follow the independent instruction contract?
Did stated confidence match observed correctness?

The result is not a universal model-intelligence score. It is a versioned measurement artifact for a declared task distribution, model release, harness and grading policy.

Learning objectives

By the end you will be able to:

distinguish task capability from access to facts;
classify cases as self-contained, parametric, evidence-backed, tool-required or unanswerable;
grade action selection separately from answer correctness;
explain why instruction following is not implied by a correct answer;
design answer, abstain and tool-use contracts before running a model;
calculate direct-answer coverage and selective accuracy;
calculate binary Brier score and fixed-bin expected calibration error;
explain what Brier and ECE each hide;
identify overconfident wrong answers explicitly;
compare in-distribution, shifted and adversarial slices;
validate case observations before aggregation;
avoid granting semantic meaning to a model’s fluent tone;
state what verbalized confidence does and does not measure;
design held-out calibration and drift checks;
turn a failure into the right intervention: model, context, tool, prompt, policy or grader.

Completion artifact

Run the example:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-evals --example capability_calibration

The eight-case fixture deliberately includes:

three in-distribution answers;
one correct abstention;
two correct tool decisions;
one confidently wrong shifted answer;
one confidently wrong answer to an unanswerable adversarial case;
one correct answer that violates its instruction contract.

Its top-level result is:

{
  "total_cases": 8,
  "task_successes": 6,
  "task_success_rate": 7500,
  "instruction_followed": 6,
  "instruction_following_rate": 7500,
  "direct_answer_opportunities": 6,
  "answered": 5,
  "correct_answers": 3,
  "direct_answer_coverage": 8333,
  "selective_accuracy": 6000,
  "overconfident_wrong_answers": 2,
  "brier_score_millionths": 365340,
  "expected_calibration_error": 4220
}

Rates use basis points: 7500 means 75.00%. The Brier score uses millionths: 365340 means 0.365340. Integer representations make serialized reports stable and avoid promising more decimal precision than the format carries.

The aggregate is intentionally uncomfortable. Its slices explain why:

Slice	Task success	Instruction following	Selective accuracy	ECE
In distribution	100.00%	66.67%	100.00%	10.67%
Shifted	66.67%	100.00%	0.00%	92.00%
Adversarial	50.00%	50.00%	0.00%	97.00%

A deploy decision based only on the in-distribution answer accuracy would miss every confidently wrong answer.

The five dimensions

1. Task capability

Capability asks whether the system can execute the transformation or decision under stated conditions:

solve a bounded reasoning problem;
extract fields from supplied text;
write compiling Rust under a declared toolchain;
classify an image according to a rubric;
align audio timestamps;
choose and invoke a tool;
abstain when evidence is insufficient.

Capability belongs to the system, not just the weight file. Prompt rendering, retrieval, tool contracts, parsers, retries, sampling, model release and resource limits all affect the observation. Report the complete release identity.

Do not infer a general capability from one success. Cases sample a distribution. “Passed these 200 versioned cases” is evidence. “Understands Rust” is an expansive conclusion that needs many task families, held-out cases, shifts and repeated runs.

2. Knowledge access

Knowledge asks whether the information required for this case was available through an allowed channel. The companion uses five explicit conditions:

pub enum KnowledgeCondition {
    SelfContained,
    Parametric,
    ProvidedEvidence,
    ExternalToolRequired,
    Unanswerable,
}

SelfContained means the case supplies all facts, as in arithmetic or a closed logic puzzle. Parametric means the pinned model is expected to contain relatively stable learned information. ProvidedEvidence means the harness supplies trusted passages, tables, images or other evidence. ExternalToolRequired means the answer depends on current, private or authoritative state that must be queried. Unanswerable means no allowed source can support an answer.

These are evaluator declarations, not model self-reports. They encode the test contract.

The boundaries matter:

A current stock price is not safely “parametric”; it requires a live source.
A private incident status is not recoverable from public model weights.
A provided document can conflict with parametric memory; the task must say which source controls.
Retrieval returning zero relevant passages does not transform an unanswerable case into a permission to guess.
A model may contain a fact but fail to retrieve it during generation.
A model can produce a correct-looking answer from an invalid source or lucky guess.

Correctness alone therefore does not prove correct knowledge use. For high-stakes or provenance critical work, also grade evidence IDs, spans, timestamps, tool traces and entailment.

3. Required action

Before inference, define the action contract:

pub enum ExpectedAction {
    Answer,
    Abstain,
    UseTool,
}

pub enum ActualAction {
    Answered,
    Abstained,
    UsedTool,
}

For this teaching artifact, the policy is deliberately strict:

Knowledge condition	Required first action
Self-contained	Answer
Parametric	Answer
Provided evidence	Answer
External tool required	Use tool
Unanswerable	Abstain

This table is a policy choice, not a law of nature. A medical workflow may require a tool even when the model probably knows the fact. A writing assistant may permit “answer with uncertainty.” An agent may need AskUser, Escalate, or RefuseUnsafeAction. Extend the enum and tests rather than smuggling those meanings into free-form strings.

The scorer defines task success from both action and answer result:

expected Answer  → actual Answered AND answer correct
expected Abstain → actual Abstained
expected UseTool → actual UsedTool

This makes correct abstention a success. It also prevents a lucky direct answer from passing a case whose contract required an authoritative tool.

“Used tool” only scores the first decision in this chapter. A production agent must additionally grade tool name, arguments, authorization, result handling, citation and final answer. Those are harness and evaluation chapters later in the book.

4. Instruction following

Instruction following asks whether output obeyed the contract:

returned the required schema;
respected a word or time limit;
used the requested language;
included source IDs;
did not expose hidden material;
stopped before a forbidden side effect;
followed precedence when instructions conflicted.

A response can be factually correct and still fail this dimension. In the example, evidence-summary gets the substance right but violates its declared format. That yields:

task success             100%
instruction following      0%

Do not use “answer correct” as a proxy for compliance. Use deterministic schema validation where possible. For semantic constraints, define a rubric, collect grader evidence, blind model identity, randomize answer order where relevant, and measure grader agreement. A model judge is another fallible component; version it and audit it against human-labeled cases.

Instruction conflicts must be represented before grading. A system should follow higher-priority policy over an injected document instruction. Marking that refusal as “did not follow the document” would reward the wrong behavior.

5. Calibration

Calibration compares confidence with empirical correctness over a set of comparable predictions. If predictions labeled 80% correct are right about 80% of the time, that group is calibrated. One correct 80% answer is not evidence of calibration; calibration is a population property.

Confidence is not:

truth;
explanation quality;
probability that every sentence is correct;
evidence that the model inspected its own weights;
a portable number across prompts, models or domains;
authorization to act.

This chapter accepts a probability attached to the binary event “the direct answer passes the case’s correctness grader.” The event definition must be stable. If one case grades exact match and another grades a vague preference, their confidence labels are not automatically comparable.

For hosted text models without useful token probabilities, a harness may ask for a verbalized probability. Research has shown that elicitation strategies can affect calibration, so record the exact prompt and treat the number as another model output—not privileged introspection. For local classifiers, confidence might instead be a calibrated function of logits. For generative answers, sequence likelihood is not automatically the probability of semantic correctness.

Step 1: validate observations before scoring

The transport-facing structure is easy to deserialize:

pub struct BehaviorObservationSpec {
    pub case_id: String,
    pub slice: EvaluationSlice,
    pub knowledge_condition: KnowledgeCondition,
    pub expected_action: ExpectedAction,
    pub actual_action: ActualAction,
    pub answer_correct: Option<bool>,
    pub instruction_followed: bool,
    pub confidence_basis_points: Option<u16>,
}

But not every combination is meaningful. The validated BehaviorObservation enforces:

bounded machine-safe case IDs;
confidence within 0..=10,000 basis points;
answered cases have correctness and confidence;
abstention/tool cases do not carry fictitious answer correctness or confidence;
knowledge condition and expected action match the declared policy.

The raw type derives Deserialize with deny_unknown_fields. The validated type does not derive Deserialize; callers must cross try_from_spec. This is the same parse-then-validate boundary used throughout the companion.

Why forbid confidence on an abstention? Because this report calibrates answer correctness. A separate system may predict “probability that abstention is appropriate” or “probability tool call will succeed,” but those are different events and deserve different fields and scorers. Reusing one number would create false precision.

Duplicate case IDs are rejected. Without stable unique IDs, baseline comparisons, failure investigation and deduplication become ambiguous.

Step 2: preserve distribution slices

The fixture labels:

pub enum EvaluationSlice {
    InDistribution,
    Shifted,
    Adversarial,
}

An in-distribution case resembles development data. A shifted case changes time, source, phrasing, length, language, modality or operating conditions. An adversarial case is intentionally chosen to induce unsafe, unsupported or contract-breaking behavior.

These labels are coarse. A real suite should use a versioned taxonomy such as:

domain=rust
difficulty=medium
language=en
freshness=live
knowledge=tool_required
attack=indirect_prompt_injection
tenant=synthetic

Keep sample counts beside rates. A 100% slice with one case is not strong evidence. Compute uncertainty intervals when making statistical claims, and never treat overlapping noisy estimates as a decisive ranking.

Avoid tuning repeatedly on the hidden test. Maintain:

development cases for iteration;
held-out evaluation cases;
adversarial red-team cases;
production shadow cases with privacy controls;
a final acceptance set whose labels are not exposed to the optimizer.

If a failure becomes a prompt example or training record, it has left the held-out set.

Step 3: measure coverage and selective accuracy together

An abstaining system can trade coverage for lower error. The companion reports:

direct-answer coverage = answered / direct-answer opportunities
selective accuracy      = correct direct answers / direct answers

Tool-required cases are excluded from direct-answer opportunities. Otherwise a system that correctly calls tools would appear to have low answer coverage.

Suppose 100 direct-answer opportunities yield:

Policy	Answered	Correct	Coverage	Selective accuracy
Always answer	100	80	100%	80%
Abstain below threshold	60	57	60%	95%
Aggressive abstention	10	10	10%	100%

The last policy is not unconditionally best. A customer-support draft tool may value coverage; an irreversible medical action may require very low residual risk. Choose the operating point from costs and constraints, then report both values.

One threshold is not enough. Sort answer candidates by a confidence signal and sweep thresholds to produce a risk–coverage curve:

risk = 1 - selective accuracy
coverage = fraction answered

This reveals whether the score meaningfully orders safer and riskier cases. A model can have decent calibration while ranking examples poorly, or good ranking while its numeric probabilities are miscalibrated.

The companion’s current fixture scores the observed operating point; a later evaluation chapter adds paired comparisons, uncertainty and policy thresholds.

Step 4: calculate Brier score

For a direct answer i, let:

p_i = stated probability that the answer is correct
y_i = 1 if the correctness grader passes, otherwise 0

The binary Brier score is:

Brier = (1/N) Σ (p_i - y_i)²

Lower is better; zero is perfect. A wrong answer at 97% confidence incurs:

(0.97 - 0)² = 0.9409

A correct answer at 97% incurs:

(0.97 - 1)² = 0.0009

The scorer calculates with basis-point integers and stores millionths:

let error = confidence_basis_points.abs_diff(outcome_basis_points);
let squared = error.pow(2);

It sums in u128, divides once, and rounds to the nearest millionth. Checked construction bounds every confidence, so the result cannot exceed one million.

Brier is a proper scoring rule for the declared binary event: in expectation, honest probabilities minimize it. But it combines calibration and resolution. Two systems with the same Brier score can have different reliability diagrams and operational value. Compare against meaningful baselines, including class prevalence and incumbent behavior, on the same cases.

Do not compare Brier scores if correctness events differ. “Exact string match,” “human judged acceptable,” and “every factual claim entailed” are different targets.

Step 5: calculate fixed-bin ECE

Expected calibration error groups predictions into confidence bins. For bin b:

gap_b = |mean_confidence_b - empirical_accuracy_b|

ECE = Σ (count_b / N) × gap_b

The example’s 90–100% bin contains three answers:

mean confidence      94.67%
empirical accuracy   33.33%
absolute gap         61.33%

That bin immediately exposes severe overconfidence.

The implementation supports 2..=100 bins. It correctly places the exact 100% endpoint in the last bin and calculates integer boundaries even when 10,000 is not divisible by the bin count. Empty bins are omitted from the report, while the configured bin count remains part of report identity.

ECE is a diagnostic, not a sufficient objective:

results depend on bin boundaries and count;
small bins have noisy empirical accuracy;
cancellation across regions can hide shape;
class imbalance can make aggregate calibration misleading;
a constant base-rate forecast can be calibrated but uninformative;
a low ECE does not guarantee safe performance at a specific decision threshold.

Always report a proper scoring rule such as Brier alongside reliability bins, accuracy, coverage, sample counts and task slices. Plotting adaptive/equal-count bins can help diagnose sparse regions, but changing binning changes the estimate.

Step 6: make dangerous errors first-class

The report counts wrong answers at or above a configured overconfidence threshold:

overconfident_wrong_answers = 2
threshold                    = 80.00%

This is not a replacement for Brier or risk-weighted evaluation. It is an operational triage list. The harness should retain case IDs and safe traces so engineers can ask:

Was required evidence absent?
Did retrieval return irrelevant material?
Did the prompt imply that guessing was rewarded?
Did a tool policy incorrectly permit direct answering?
Did a parser attach confidence to the wrong candidate?
Did a grader reject a semantically correct response?
Is the failure isolated to one model release or slice?

Never log private prompts merely to make debugging convenient. Store content-free identifiers and approved redacted artifacts according to the data policy established in the production chapters.

Interpreting the runnable report

The fixture makes several distinctions visible:

In-distribution

All three answers are correct, so task success is 100%. One violates an instruction, so compliance is 66.67%. Answer confidences are high and errors are small, yielding a low Brier score.

Intervention: improve the structured-output harness or contract training. Replacing the model for “knowledge” is not the first hypothesis.

Shifted

The system correctly abstains on a private fact and correctly uses a live-weather tool. It also answers a stale API question incorrectly at 92% confidence. Task success is therefore 66.67%, but selective accuracy among direct answers is zero.

Intervention: strengthen evidence precedence, freshness metadata and “provided evidence required” checks. The success of tool routing should not hide the stale-evidence failure.

Adversarial

The system safely chooses a tool in an injection case, but confidently accepts a false premise in an unanswerable case. Task success and instruction following both fall to 50%.

Intervention: add false-premise and absence-of-evidence training/eval cases, enforce an abstention contract, and test whether the harness can distinguish “no source” from “source supports answer.”

Knowledge breakdown

Tool-required cases score 100% while unanswerable cases score 50%. This points away from a general tool-routing failure and toward recognition of unknowable claims. Aggregate task success alone would not localize that.

A production evaluation protocol

1. Freeze the contract

Version:

task definition and acceptable actions;
model artifact/provider release;
system/developer/user prompt renderer;
tool schema and policy;
retrieval corpus snapshot and query logic;
sampling settings and repetition count;
correctness and instruction graders;
confidence elicitation method;
score configuration and thresholds.

Changing any of these can change the measured system.

2. Build cases from a causal taxonomy

Do not create 1,000 paraphrases of one easy fact and call it comprehensive. Cross important axes:

task × knowledge source × difficulty × freshness × language
× modality × risk × instruction conflict × attack × answerability

Deduplicate by semantics as well as bytes. Track case provenance and licensing. Keep sensitive production examples isolated and minimized.

3. Grade observable claims

Prefer deterministic checks:

JSON Schema for structure;
compiler/tests for code;
exact source IDs for provenance;
numerical tolerances declared in advance;
policy engine decisions for tool authorization;
byte/duration/dimension checks for media.

When human judgment is required, use a rubric and multiple blinded annotators for a sample. Measure disagreement. When a model judge is used, keep its raw judgment evidence, version and known bias tests.

4. Repeat stochastic runs

One sample estimates neither success probability nor output diversity. Choose repetitions based on variance and cost. Record seed/draw stream where the runtime supports it, while retaining the reproducibility limits from the sampling chapter.

Aggregate by case first so frequently repeated easy cases do not dominate. Report pass@k only when the product is actually allowed k attempts; otherwise it exaggerates deployed performance.

5. Fit calibration only on calibration data

Temperature scaling, isotonic regression or another calibration map must be fitted on a separate calibration split, then evaluated on held-out data. Fitting and reporting on the same cases is optimistic.

Calibration can drift when:

model weights or provider aliases change;
prompts change;
retrieval changes the case mix;
confidence elicitation wording changes;
users become more adversarial;
the correctness grader changes;
the deployment shifts domains or languages.

Pin the calibration-map version to the complete system release. Monitor Brier, reliability bins and coverage by meaningful slice. Recalibrate only through a reviewed release.

6. Choose policy from consequences

Calibration estimates probability; decision policy combines probability with costs:

expected loss(answer)   = P(wrong) × cost(wrong)
expected loss(abstain)  = cost(abstain)
expected loss(use_tool) = tool cost + latency + residual failure risk

The lowest-loss action can differ by tenant and task. A 90% answer may be acceptable for brainstorming and unacceptable for deleting data. Authorization and safety constraints still override expected utility.

7. Gate regressions, not just averages

A candidate must not buy a tiny aggregate gain by regressing a critical slice. Example release policy:

overall task success:        no worse than -0.5 percentage points
critical tool policy:        zero unauthorized direct answers
unanswerable slice:          overconfident wrong count must not increase
instruction compliance:     ≥ 99.5% for machine-consumed output
Brier on held-out answers:   no worse than baseline with uncertainty reported
latency/cost:                within declared budget

Thresholds are product decisions. The important property is that they are declared before seeing the candidate’s final result.

Common category errors

“The model knew it because it answered correctly”

It may have guessed, copied a misleading source or exploited leakage. Inspect provenance and create counterfactual cases.

“The model is incapable because the answer was wrong”

The model may have lacked current evidence, received a truncated prompt, been routed to the wrong tool or lost output in parsing. Diagnose the system stage.

“It followed instructions because the answer was useful”

Usefulness does not prove schema, policy or precedence compliance. Grade those contracts directly.

“It said 95%, so the answer is probably right”

Only held-out empirical calibration for this event and system gives that number operational meaning. Fluency and confidence wording are not guarantees.

“ECE is low, therefore probabilities are good”

ECE depends on binning and can reward unsharp forecasts. Report Brier, reliability shape, resolution, coverage and task performance.

“Abstention improves quality”

It can trivially improve selective accuracy by answering almost nothing. Report the full risk–coverage behavior and product utility.

“The benchmark is fixed, so progress is objective”

Repeated optimization leaks information. Cases become stale, products change, and public benchmarks can enter training corpora. Maintain hidden, refreshed and production-relevant sets.

Exercises

Exercise 1: diagnose without a scalar

A model answers 90/100 answerable cases correctly, guesses on 8/10 unanswerable cases, and obeys the JSON schema on 95/110 total cases.

Compute separate answer accuracy, unanswerable action success and instruction-following rate. Explain why “90% accurate” is misleading.

Solution

Answerable accuracy is 90%. Unanswerable action success is 2/10 = 20% if only the two non-guesses abstained correctly. Instruction following is 95/110 = 86.36%. The system performs ordinary answers reasonably but recognizes epistemic boundaries poorly and breaks a machine contract often.

Exercise 2: Brier arithmetic

Three answers have (confidence, correct) values (0.9, 1), (0.8, 0), (0.6, 1). Calculate Brier.

Solution

[(0.9 - 1)² + (0.8 - 0)² + (0.6 - 1)²] / 3
= (0.01 + 0.64 + 0.16) / 3
= 0.27

Exercise 3: add an action

Add AskUser to expected/actual actions. Decide which knowledge conditions may require it. Add constructor invariants, task-success logic, JSON output and tests for an underspecified request.

Do not reinterpret Abstain as “ask a question”; the user experience and continuation semantics are different.

Exercise 4: sweep coverage

Extend the example to sort direct answers by confidence and emit every unique threshold’s coverage and risk. Test stable ordering for equal confidence. Then show a counterexample where confidence is well calibrated in aggregate but ranks the two risk groups poorly.

Exercise 5: audit the grader

Create 30 responses with human correctness labels. Run the automated grader blind, build a confusion matrix and inspect disagreements. Decide whether grader false positives or false negatives are more costly for the product.

Exercise 6: multimodal confidence

For an image measurement task, define distinct events:

object present;
bounding box meets IoU threshold;
OCR text exactly correct;
final composed answer correct.

Explain why one confidence number should not be copied across all four.

Completion checklist

You are done when you can demonstrate:

Sources and further study

Guo, Pleiss, Sun, and Weinberger, On Calibration of Modern Neural Networks: confidence calibration, reliability diagrams, ECE and temperature scaling.
Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation: the theory and interpretation of proper scoring rules.
El-Yaniv and Wiener, On the Foundations of Noise-free Selective Classification: the risk–coverage tradeoff in selective prediction.
Liang et al., Holistic Evaluation of Language Models: multi-metric evaluation across scenarios, including calibration and robustness.
Tian et al., Just Ask for Calibration: experiments on eliciting verbalized confidence from instruction-tuned language models.
Lin, Hilton, and Evans, TruthfulQA: evidence that imitation of human text can reproduce common false beliefs, so scale and fluency are not truth guarantees.
NIST, Generative AI Profile (NIST AI 600-1): operational risk guidance including confidently stated erroneous content.

What this chapter does not claim

The scorer does not prove a model “knows what it knows.” It does not derive semantic confidence from logits, calibrate free-form generations automatically, validate tool results, estimate statistical uncertainty, fit a calibration map, or establish safe deployment. It demonstrates a minimum honest measurement boundary: capability, knowledge conditions, action choice, instruction following, confidence and distribution shift must not be collapsed into one score.

The next chapter separates text generation, embedding, reranking and classification capabilities so provider adapters expose what they actually implement instead of one misleading universal generate interface.