Model vs Harness vs Product

An AI product is often discussed as if the model were the whole system. That makes engineering decisions blurry. Use four layers instead.

The four layers

1. Model

The model maps context to a distribution over outputs. It contains learned capability, knowledge, biases, and limitations. Parameters such as temperature or maximum output length affect sampling, but they do not turn the model into a reliable product.

2. Harness

The harness constructs and controls a run:

• task specification and instructions;
• context selection and compression;
• available tools and their schemas;
• permissions and action policy;
• memory and state;
• retries, stopping rules, and budgets;
• output schemas and validators;
• trace capture and evaluation hooks;
• model selection and fallback.

When people say “prompt engineering,” they often mean one small part of harness engineering.

3. Product

The product decides why the run exists and what happens next. It owns user intent, interaction design, authorization, persistence, review, and the consequences of an action.

4. Environment

The environment is the world the system reads and changes: a repository, browser, ticket queue, database, media library, deployment, or filesystem. It supplies observations and receives effects.

user intent
    │
    ▼
 product policy ──▶ harness ──▶ model
    │                  │          │
    │                  ├─ tools ◀─┘
    │                  ├─ trace
    │                  └─ verifier
    ▼
 environment ◀──── authorized effects

This separation is practical. If an answer lacks a fact, change retrieval or the model—not the button label. If the correct answer causes an unsafe write, fix product authorization or tool policy—not the temperature.

What a strong harness can and cannot do

A strong harness can:

• narrow an open problem into smaller decisions;
• supply current or private knowledge;
• replace arithmetic or lookup with deterministic tools;
• constrain syntax;
• verify observable properties;
• ask for repair when a specific invariant fails;
• route uncertain cases to a stronger model or human;
• spend extra inference only where it improves the measured task.

It cannot guarantee:

• knowledge that is absent from both model and context;
• reasoning beyond the model on every distribution;
• truth when no trustworthy evidence exists;
• semantic correctness merely because JSON parsed;
• safe behavior when dangerous tools have broad ambient authority.

The right claim is therefore:

A harness can make a model substantially more useful and reliable on a bounded task. The size of the gain is an empirical result, not a slogan.

Workflows and agents

A workflow follows a predefined graph. An agent chooses some of its own next steps. Both are useful.

Use a workflow when:

• the stages are known;
• the cost of variation is high;
• each step has a clear contract;
• deterministic code can make the decision.

Use an agent when:

• the path depends on intermediate discoveries;
• the environment is too varied for a fixed graph;
• the model can choose among well-scoped tools;
• you have a stopping rule and an outcome verifier.

Anthropic’s building effective agents guidance makes the same architectural distinction. Start with the simplest composition that works; autonomy is a cost and risk budget, not a badge of sophistication.

The harness as a state machine

Represent a run as explicit transitions:

enum RunState {
    Preparing(Task),
    Ready(PreparedRun),
    Calling(ModelRequest),
    Acting(PendingToolCall),
    Verifying(Candidate),
    Completed(VerifiedOutput),
    Failed(RunFailure),
}

This is more than code style. Completed(VerifiedOutput) communicates a stronger fact than Completed(String). PendingToolCall can carry the required permission. Invalid transitions can be impossible to express.

An agent loop then becomes ordinary control flow:

while budget.can_continue() {
    let observation = model.respond(&context).await?;

    match policy.decide(observation)? {
        Decision::Answer(candidate) => return verifier.finish(candidate).await,
        Decision::Call(tool_call) => {
            let result = tools.execute(authorization, tool_call).await?;
            context.observe(result);
        }
        Decision::Repair(feedback) => context.add_feedback(feedback),
        Decision::Escalate(reason) => return Err(RunFailure::NeedsHuman(reason)),
    }
}

Err(RunFailure::BudgetExhausted)

Every noun in this loop becomes a replaceable Rust boundary and an evaluation surface.

Evaluate the system you ship

An agent evaluation does not measure the model in isolation. It measures the model plus instructions, tools, loop, environment, and graders. Anthropic’s guide to agent evals calls the component that runs this end-to-end process the evaluation harness.

Keep two concepts distinct in this book:

• runtime harness — performs real tasks;
• evaluation harness — creates controlled tasks, runs the runtime harness, and grades the result.

They share types and trace formats, but the evaluation harness must not silently depend on production state.

The first design exercise

Choose an AI feature you use. Draw its four layers. Then answer:

1. Which facts come from the model, retrieval, tools, and user?
2. Which actions can change the environment?
3. Where is authorization checked?
4. What observable result proves success?
5. Which failures should retry, stop, or escalate?
6. What would a smaller model need in order to do this narrow job?

If those answers are unclear, the system is not ready for another prompt iteration. It needs a better design.