Verify, Repair, and Route

The cheapest acceptable system is often not one model. It is a path:

small model → deterministic verification → targeted repair
           ↘ uncertainty or repeated failure → stronger model → review

Verification must be independent enough

Asking the same model “Are you sure?” adds little evidence. Prefer checks grounded outside the original generation:

• compiler and tests for code;
• schema and invariant validation;
• citation lookup;
• database or API state;
• image geometry;
• audio/video timestamp bounds;
• a different model with a focused rubric;
• human review for subjective or consequential cases.

Verification is strongest when success is observable.

Targeted repair

Return exact violations:

struct RepairRequest {
    candidate: CandidateRef,
    violations: Vec<Violation>,
    allowed_changes: Vec<JsonPath>,
    attempts_remaining: u8,
}

Allow the model to patch only invalid fields when possible. This reduces token use and protects parts that already passed.

Stop when:

• the output passes;
• attempts are exhausted;
• the same violation repeats;
• the candidate becomes worse on a guardrail;
• a stronger route is cheaper than another repair.

Routing signals

Route using features available before or during a run:

• modality and input size;
• language or domain;
• privacy policy;
• local model residency;
• retrieval confidence;
• small-model log probability or self-reported uncertainty, if calibrated;
• disagreement between extractors;
• verifier violations;
• time and cost remaining;
• risk class.

Avoid routing solely on a model saying “this is hard.” Calibrate signals against actual failures.

A typed routing decision

enum RouteDecision {
    Use(ModelId),
    Cascade { first: ModelId, fallback: ModelId },
    ParallelJudge { candidates: Vec<ModelId>, judge: ModelId },
    HumanReview { reason: ReviewReason },
}

Persist the reason. A cost regression is easier to diagnose when traces show which signal escalated.

Cascades are an empirical design

The FrugalGPT paper studied model cascades that learn which model combinations to use for cost and accuracy. Its reported gains are task- and model-specific, but the principle is durable: heterogeneous models create a routing opportunity.

Build your cascade from local data:

1. run all candidate models on a representative set;
2. identify where the cheaper model succeeds;
3. find signals that predict its failures;
4. define the route and fallback;
5. evaluate the cascade end to end;
6. monitor distribution drift.

Judges can become the bottleneck

If a frontier judge evaluates every small-model answer, the system may cost more than calling the frontier model directly. Use deterministic checks first and judge only ambiguous cases.

Model graders also have bias. Calibrate them against human labels and inspect disagreement slices.

Practice

Build an extraction cascade:

• local OCR;
• small text model producing a typed record;
• deterministic field validators;
• small-model repair;
• large vision model fallback only when OCR or validation confidence is low.

Report exact accuracy, fallback rate, mean cost, p95 latency, and severe errors. Tune the threshold on a development set and make the final decision on held-out cases.