Decompose Work for Smaller Models

A smaller model often fails because the system gives it too many simultaneous responsibilities: discover the task, find evidence, reason, format, check policy, and decide when to stop.

Decomposition removes responsibilities from the model.

Start from the completion condition

Suppose Signal Room must produce a supported incident timeline. Break it into observable stages:

normalize evidence
  → extract timestamped events
  → link duplicate or related events
  → propose causal edges
  → test edges against chronology and evidence
  → identify earliest supported anomaly
  → render report with citations and unknowns

Deterministic code handles normalization, timestamp ordering, citation existence, and graph validation. Models handle ambiguous extraction and causal hypotheses.

Good module boundaries

A module should have:

• one decision;
• minimal context;
• a narrow output schema;
• a local grader;
• a clear failure or abstention;
• no authority it does not need.

Bad:

Read this repository and fix the bug.

Better:

1. Classify failing test by subsystem.
2. Retrieve symbols and recent changes for that subsystem.
3. Propose up to three hypotheses with evidence.
4. Choose one experiment that discriminates between them.
5. Run the authorized experiment.
6. Update the hypothesis scores.
7. Propose a patch.
8. Compile, test, lint, and inspect the diff.

An agent may still choose which hypothesis to test, but the harness owns the scientific structure.

Prefer deterministic transforms

Do not spend model capability on:

• sorting timestamps;
• validating paths;
• parsing JSON;
• arithmetic;
• exact lookup;
• schema conversion;
• checking whether tests pass;
• cropping a known image region;
• extracting a video frame at a known timestamp.

Tools make the model’s task smaller and the result easier to evaluate.

Use intermediate representations

Free-form summaries lose information between stages. Define an intermediate representation:

struct Event {
    id: EventId,
    time: TimeRange,
    actor: Option<String>,
    action: String,
    object: Option<String>,
    evidence: Vec<Citation>,
    uncertainty: Vec<String>,
}

Later modules operate on events rather than re-reading raw media. They can return to source evidence when needed.

Decomposition can also hurt

Each boundary can:

• discard useful context;
• introduce latency;
• multiply calls;
• create error propagation;
• make global reasoning harder.

Compare against a simple baseline. Merge stages that repeatedly need the same context or whose local graders do not predict end-to-end quality.

A difficulty-aware graph

Not every case needs every stage:

input
 ├─ easy + high confidence → extract → deterministic verify → done
 ├─ uncertain evidence → retrieve more → extract → verify
 └─ conflicting evidence → stronger synthesis model → human review

The small model handles the high-volume narrow path. Complexity is spent only on evidence of difficulty.

Practice

Take “critique this product demo video” and design modules for:

• transcript cleanup;
• scene segmentation;
• claim extraction;
• visual evidence matching;
• pacing metrics;
• contradiction detection;
• final critique.

For each, identify deterministic inputs, model decision, schema, grader, and abstention condition. Then compare the modular pipeline with one end-to-end prompt on ten labeled videos.