fine-tuning-techniques
Fine-Tuning Techniques
📂 fine-tuning
FINE-TUNING GUIDE#
Fine-Tuning Techniques
Make two separate choices: what learning objective matches your data, and how much of the model to update. No method wins across every model, task, and hardware stack.
1. Choose the learning objective#
Supervised fine-tuning (SFT)#
Data: Input and desired-output examples
Teach a stable task, response structure, style, or tool-calling pattern.
Watch: The model may imitate errors and biases in demonstrations; unrelated behavior can regress.
Preference optimization#
Data: Preferred and non-preferred responses, or equivalent feedback
Shift choices where quality is easier to compare than to specify as one ideal answer.
Watch: Label policy, annotator agreement, and held-out evaluation matter as much as the algorithm.
Reinforcement fine-tuning#
Data: Prompts plus a reliable grader or reward signal
Optimize outcomes that can be scored consistently across generated trajectories.
Watch: Reward hacking, grader bias, instability, and higher operational complexity require extra controls.
2. Choose the parameter-update method#
Full-parameter tuning#
Updates the base weights and offers the most flexibility, with the largest memory, storage, and rollback burden.
Consider only when adapter methods miss a measured target and the infrastructure and data justify it.
LoRA / adapters#
Freezes the base model and trains smaller update modules, often under one percent of the parameter count. Artifacts are compact and multiple adapters can share one base.
A sensible first experiment for many transformer workloads; rank and target modules still need empirical tuning.
QLoRA#
Backpropagates through a quantized frozen base into LoRA adapters, reducing base-weight memory at potential compatibility and throughput costs.
Useful when memory is the binding constraint; validate quantization support and final serving quality.
Variants are hypotheses, not upgrades#
Adapter variants#
DoRA, AdaLoRA, LoRA+, and other variants change parameterization, allocation, or optimization. Try one only when its implementation supports your model and it addresses a measured limitation of the simpler baseline.
Quantization-aware training#
QAT targets quality after low-precision deployment. It is different from QLoRA, which primarily uses a quantized frozen base to reduce training memory. Validate the exported format on the intended runtime.