LoRA, QLoRA, and Preference Optimization
Fine-tuning changes weights. The harness still owns context, tools, policy, verification, and evaluation.
Supervised fine-tuning
Section titled “Supervised fine-tuning”SFT trains the model to imitate desired outputs or trajectories. It is useful for:
- consistent formats and terminology;
- narrow classification or extraction;
- tool-selection patterns;
- domain-specific response behavior;
- compressing a successful multi-call strategy into fewer calls.
Its quality is bounded by example correctness and coverage.
Low-Rank Adaptation learns small low-rank weight updates while keeping the base model frozen. The original LoRA paper introduced the approach to reduce trainable parameters and memory.
Operationally, you ship:
- exact base model ID and hash;
- adapter weights;
- adapter configuration and rank;
- tokenizer and chat template;
- training dataset version;
- training code and hyperparameters;
- eval report.
An adapter without its base artifact is incomplete.
QLoRA backpropagates through a frozen quantized model into LoRA adapters, substantially reducing memory requirements. It makes experimentation more accessible, but it does not remove dataset, evaluation, or deployment work.
Evaluate the actual inference stack. Training with one representation and serving with another quantization can change quality.
Preference optimization
Section titled “Preference optimization”Preference data contains a chosen and rejected response for the same input. Direct Preference Optimization methods train the model toward those comparisons without a separate reward-model RL pipeline.
Use preference optimization when the distinction is genuinely comparative:
- which critique is more actionable;
- which generated caption better matches an image;
- which tool plan is safer and more efficient;
- which response follows the product’s tone without losing facts.
Preference labels need a rubric. “I like A” is weak data unless liking is the product objective.
A disciplined training run
Section titled “A disciplined training run”- define the target behavior and primary eval;
- freeze untuned baseline;
- inspect and version data;
- run a small training sweep;
- evaluate checkpoints on development data;
- select once;
- evaluate on held-out and safety slices;
- compare cost and latency;
- canary with rollback;
- monitor drift and new failure clusters.
Multimodal tuning
Section titled “Multimodal tuning”Multimodal adapters may target:
- a vision-language model’s domain vocabulary;
- image embeddings for specialized retrieval;
- speech recognition for acoustic or vocabulary domains;
- text-to-speech voice or prosody, with consent;
- diffusion style or subject adapters, with rights and provenance.
The evaluation must match the modality. A text-only judge cannot establish visual identity, temporal coherence, or audio quality by itself.
Watch for regressions
Section titled “Watch for regressions”Track:
- held-out task quality;
- general capability slices;
- refusal and safety behavior;
- hallucination and citation support;
- tool-call validity;
- calibration;
- latency and memory;
- sensitive data memorization tests.
If a tuned small model improves a narrow task but becomes worse at recognizing uncertainty, the harness may need stronger escalation thresholds.
Practice
Section titled “Practice”Take the Forge extraction task:
- collect verified traces from the decomposed harness;
- create SFT examples for the small model;
- train a LoRA or QLoRA adapter;
- compare untuned model + rich harness against tuned model + simpler harness;
- compare both against the routed frontier baseline.
The winning system is the one on the quality/cost/latency/risk frontier, not the one with the most interesting training run.