Spend Test-Time Compute Deliberately

Some tasks improve when the system spends more inference at run time. More tokens or candidates are not automatically better; they create a search problem and a cost problem.

Four ways to spend compute

Longer internal reasoning

Give one model more room to reason. This is simple but difficult to inspect and may plateau.

Multiple candidates

Generate diverse candidates, then select with a verifier or judge.

Search over actions

Explore hypotheses, tool calls, or plans and prune weak branches.

Specialized roles

Use separate proposer, critic, verifier, or synthesizer calls. Role labels alone add little; each role needs different evidence or a distinct rubric.

Budget the search

struct SearchBudget {
    max_candidates: usize,
    max_depth: usize,
    max_model_calls: u32,
    max_tool_calls: u32,
    deadline: Instant,
    max_cost_micros: u64,
}

A search strategy must work inside all limits. Reserve budget for final verification.

Best-of-N needs a trustworthy selector

If the selector cannot distinguish good candidates, generating more creates confident noise.

Use:

• exact outcome tests;
• constraint satisfaction;
• grounded citation checks;
• calibrated reward or judge models;
• human preference for subjective media.

Measure pass@N, selection accuracy, total cost, and the rate at which selection chooses a worse candidate than the first.

Search where branching has value

For incident diagnosis, diverse causal hypotheses may help. For extracting an invoice number, five reasoning branches are wasteful.

Good candidates for extra compute:

• planning with several plausible routes;
• code repair with executable tests;
• conflicting multimodal evidence;
• subjective generation with an explicit preference model;
• high-value decisions where a human will review the finalists.

Early stopping

Stop when:

• an objective verifier passes;
• additional candidates are near-duplicates;
• confidence is calibrated above a threshold;
• the expected value of another attempt is below its cost;
• the remaining budget cannot complete verification;
• the task requires human judgment.

if verifier.accepts(&candidate)
    || diversity.has_collapsed()
    || budget.cannot_afford(final_verification_cost)
{
    break;
}

Small models as fast search operators

A small model can generate hypotheses, queries, crops, or candidate transforms cheaply. A deterministic or stronger verifier then evaluates them.

Examples:

• propose code locations; compile and test patches;
• propose image regions; OCR and compare against target fields;
• propose video segment boundaries; score coverage and redundancy;
• propose retrieval queries; measure evidence recall;
• propose tool plans; simulate permissions and cost before execution.

This works because the environment supplies feedback stronger than the model’s self-assessment.

Practice

For a repository bug, compare:

1. one large-model attempt;
2. four small-model patch candidates with test selection;
3. a small-model hypothesis search with tool observations and one final large-model synthesis.

Use a fixed wall-clock and cost budget. Report solved cases, unsafe changes, attempts, and time to first passing patch. “More calls” is not the objective; verified task completion is.