Fine-Tune, Retrieve, or Redesign?

Fine-tuning is an intervention, not a milestone. Begin with the failure.

Diagnose the missing ingredient

Failure	First intervention
Current or private facts are missing	retrieval or a tool
Output syntax is inconsistent	constrained structured output
Arithmetic or exact lookup fails	deterministic tool
The task is too broad	decomposition
Good answers exist but are inconsistent	examples, optimization, or fine-tuning
Style or terminology is wrong	examples or SFT/LoRA
Preferences are subtle	preference data and DPO-like methods
Base capability is absent	stronger base model, tools, or product limit
Latency/cost is too high	routing, distillation, quantization, or smaller model

Do not train facts that change daily. Retrieve them.

Establish the untuned baseline

Freeze:

• base model artifact;
• harness and prompt versions;
• development and held-out evals;
• inference parameters;
• quality, cost, latency, and safety metrics.

Training without a baseline can produce a model that feels more specialized while regressing important slices.

Data is the actual product

Every example needs:

• task and input;
• desired behavior;
• provenance and permission;
• quality status;
• slice tags;
• deduplication identity;
• relationship to eval cases;
• reason it belongs.

Separate:

• training;
• development;
• held-out test;
• grader calibration;
• red-team cases.

Near-duplicates across splits create misleading scores.

Harvest traces carefully

Harness traces are valuable because they contain failures, repairs, tool calls, and verified outcomes. They are also dangerous: they can contain secrets, user data, incorrect model text, and grader errors.

Create a curation pipeline:

candidate traces
  → policy and consent filter
  → redact
  → deduplicate
  → verify outcome
  → classify behavior
  → human review sample
  → dataset version

Never train directly on every “successful” production run. Success may be mislabeled.

Distillation

Use a stronger teacher or successful harness to generate demonstrations for a smaller student. Retain:

• the original task distribution;
• verified final answer;
• intermediate rationale only if appropriate and safe;
• tool trajectory when teaching tool behavior;
• teacher and harness versions.

The student should be evaluated independently on held-out cases. Distillation can transfer a task strategy; it does not guarantee general frontier capability.

Synthetic data with adversarial variation

Generate variations around real behavior:

• different surface wording;
• missing or conflicting evidence;
• alternate media quality;
• boundary lengths;
• valid refusals;
• tool failures;
• injection attempts.

Use deterministic validators and human sampling. Synthetic volume cannot rescue a vague label.

Decision gate

Fine-tune only when:

• the desired behavior is stable and repeated;
• enough high-quality examples exist;
• the baseline failure is understood;
• harness-only interventions were measured;
• deployment and rollback are feasible;
• the expected volume justifies training and model operations.

Otherwise, improve the harness.