Fine-Tuning Guide — Agentic Design

Overview#

Plan, train, evaluate, and operate model adaptations with explicit trade-offs and safeguards.

What Fine-Tuning Changes#

Fine-tuning updates some or all of a pretrained model's parameters using task examples or preference feedback. It can improve measured behavior (output structure, tone, task accuracy) but does not automatically add fresh knowledge, eliminate hallucinations, or make a model safe.

Good Reasons to Test Fine-Tuning#

• A stable, repeated behavior resists prompt-only fixes
• You have representative examples and a trustworthy evaluation set
• Shorter prompts or a smaller model may improve serving economics
• The chosen checkpoint and artifacts can be deployed and governed

Reasons to Use Another Approach#

• Use retrieval or tools for changing or attributable knowledge
• Improve prompts when the desired behavior fits clear instructions
• Fix the product or data pipeline when the model lacks required context
• Delay training when data rights, quality, or evaluation are unresolved

Release-Oriented Workflow#

1. Write the product goal, acceptance rubric, and failure budget
2. Create a sealed holdout from production-shaped cases
3. Measure prompt-only, few-shot, retrieval, and tool-use baselines
4. Select a licensed base model and smallest practical adaptation method
5. Run a short, reproducible pilot and inspect learning curves
6. Compare quality, regressions, latency, throughput, and total serving cost
7. Canary the winning candidate with monitoring and tested rollback

Minimum Comparison Report#

• Base checkpoint, prompt, retrieval, and tuned candidate on same holdout
• Per-slice task results with sample counts
• Safety and base-capability regressions with release thresholds
• Training configuration, seed, code revision, data revision, artifact hashes
• Measured latency, throughput, memory, and cost on serving stack
• Named owner, canary plan, monitoring signals, rollback procedure