Fine-Tuning Frameworks

PEFT — Parameter-Efficient Fine-Tuning (HuggingFace)

• URL: github.comhuggingfacepeft
• Implemented methods: LoRA, QLoRA, IA3, Prefix Tuning, Prompt Tuning, AdaLoRA, LLaMA-Adapter

LoRA — Low-Rank Adaptation

• arXiv: 2106.09685 (Hu et al., Microsoft, 2021)
• Mechanism: Adds low-rank A × B matrices alongside the frozen weight matrices
  • W' = W + AB where A ∈ R^{d×r}, B ∈ R^{r×k}, r << d,k
• Trainable parameters: 0.1–1% of the model's parameters
• Typical rank: r8 to r64
• target_modules: qproj, vproj (minimum); add kproj, oproj, gate_proj for more expressiveness
• Fusion: merge_and_unload() merges LoRA into the weights → no overhead at inference

QLoRA — Quantized LoRA

• arXiv: 2305.14314 (Dettmers et al., UW, 2023)
• Mechanism: Base model in NF4 (4-bit) frozen + LoRA in BF16
• Double Quantization: Also quantizes the scale factors → ~0.37 additional bits per parameter
• Paged Optimizers: Optimizer states in DRAM, brought to the GPU as needed
• Result: Fine-tunes of 65B on 1 GPU of 48GB; 70B on 2× RTX 4090

AdaLoRA

• arXiv: 2303.10512
• Improvement over LoRA: Allocates rank adaptively per layer (SVD-based)
• Result: Better performance with the same number of parameters

TRL — Transformer Reinforcement Learning (HuggingFace)

• URL: github.comhuggingfacetrl
• Methods: SFT, DPO, PPO, GRPO, KTO, ORPO, SimPO, RewardTrainer
• Design: High-level wrappers over PyTorch + PEFT + Accelerate

from trl import SFTTrainer, DPOTrainer, GRPOTrainer

DPOTrainer: Pairs (prompt, chosen, rejected) → automatic DPO loss GRPOTrainer: Group relative policy optimization for RLVR RewardTrainer: Trains a preference reward model

Axolotl

• URL: github.comOpenAccess-AI-Collectiveaxolotl
• Philosophy: Config-driven (YAML) — fine-tuning without code
• Supports: LoRA, QLoRA, Full fine-tuning, FlashAttention, FSDP, DeepSpeed
• Formats: Alpaca, ShareGPT, Dolma, OASST, custom
• Config example:

base_model: Qwen/Qwen2.5-Coder-32B-Instruct
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_target_modules: [q_proj, v_proj, k_proj, o_proj]
datasets:
  - path: koder/code-review-pairs
    type: chat_template

LLaMA-Factory

• URL: github.comhiyougaLLaMA-Factory
• Origin: Peking University
• Features: WebUI for fine-tuning without code; support for 100+ models
• Methods: SFT, DPO, PPO, GRPO, ORPO, SimPO, RM
• Quantization: GPTQ, AWQ, NF4 integrated
• When to use: Those who want a WebUI; rapid experimentation

Unsloth

• URL: github.comunslothaiunsloth
• Focus: Much faster fine-tuning (~2–5×) with less memory
• Mechanism: Custom Triton kernels for LoRA + backward pass; customized gemma
• Support: Llama, Mistral, Qwen, Phi, Gemma; LoRA/QLoRA
• Limitation: NVIDIA GPU only; no DDP/FSDP (single GPU)
• When to use: Fast fine-tuning on 1 GPU; prototyping

LitGPT (Lightning AI)

• URL: github.comLightning-AIlitgpt
• Base: PyTorch Lightning
• Features: Full fine-tuning, LoRA, Adapter; pre-training; inference
• Design: Modular and readable; good for research
• Models: Llama, Mistral, Falcon, Phi, Gemma, Qwen integrated

FastChat

• URL: github.comlm-sysFastChat
• Origin: LMSYS (Berkeley)
• Focus: Serving fine-tuned models + evaluation with chatbot arena
• Features: Vicuna training script; OpenAI-compatible API; multi-model serving
• When to use: Serving fine-tuned models in production

ms-swift (ModelScope, Alibaba)

• URL: github.commodelscopems-swift
• Focus: Fine-tuning of Qwen, Qwen2.5-Coder, and other Alibaba models
• Integration: ModelScope Hub (alternative to the HuggingFace Hub)
• When to use: Models from the Qwen family (recommended by Alibaba)

torchtune (PyTorch)

• URL: github.compytorchtorchtune
• Origin: Meta / PyTorch Team (2024)
• Design: Native PyTorch; no extra dependencies; composable
• Methods: Full fine-tuning, LoRA, QLoRA
• When to use: Full control; integration with the pure PyTorch ecosystem

Comparison of Fine-Tuning Frameworks

Recommended Fine-Tuning Pipeline for Kode

1. Data preparation
   └── koder/code-datasets (instruction/code pairs)
   └── Verification: executable code passing tests

2. Initial SFT (Axolotl)
   ├── Base: Qwen2.5-Coder-32B-Instruct
   ├── Method: QLoRA r=32, target_modules=all_linear
   ├── Data: 50K–200K high-quality pairs
   └── Hardware: 2× RTX 4090 with FSDP

3. Preference Optimization (TRL)
   ├── Method: DPO
   ├── Data: Pairs (accepted/rejected) of suggestions
   └── Base: model from step 2

4. RLVR with verifiable builds (TRL GRPO)
   ├── Reward: build pass/fail + test pass/fail
   └── Base: model from step 3

Code Fine-Tuning Data

• CommitPack: diff + commit message → code review
• The Stack v2: additional code pre-training
• SWE-bench Train: issues/PRs for agent training
• Synthetic data: generate, via Claude/GPT-4o, code-review pairs specific to Koder's patterns