Day 28 — Fine-Tune or Fix the System?

Today Model Lab decides whether tuning is the responsible next lever.

Diagnose the failure class

• Missing or changing facts usually need retrieval.
• Exact arithmetic or external state usually needs a tool.
• Invalid structure usually needs schema and validation.
• A task that is too broad may need decomposition.
• Rare hard cases may need routing.
• Consistent behavior/style failure on well-specified, well-evidenced cases may justify tuning.

Do not fine-tune knowledge you can retrieve or policy you should enforce in code.

Build a governed dataset

Each example records source, rights/consent, task version, evidence, desired output, validation result, split assignment, and contamination checks. Deduplicate before splitting so near-identical examples do not leak into evaluation.

Design the experiment

Freeze:

1. base model release;
2. untuned harness baseline;
3. train/validation/test split;
4. SFT or LoRA/QLoRA configuration;
5. evaluation and safety gates;
6. inference adapter identity;
7. rollback rule.

Compare tuned model with the untuned model under the same harness. Otherwise you cannot attribute the change.

Preference optimization adds another risk: labelers may reward confident style over factual support. Audit preference pairs and grader agreement before training.

Completion proof

Write a decision memo for one failure: evidence showing earlier levers were tried, expected tuning benefit, data rights, baseline, stop rule, safety slices, and rollback. “We have data” is not enough.

Deep references: Fine-tune, retrieve, or redesign? and LoRA, QLoRA, and DPO.

Finish with Project 6 Revision →.