DPO vs KTO vs ORPO: Preference Fine-Tuning for Local LLMs

DPO, ORPO and KTO all align a model to preferences without RLHF's separate reward model and PPO loop. They differ in the data they need and the number of training stages. Quick answer: use DPO if you already have an SFT model and paired preferences, ORPO if you are starting from a base model with paired preferences, and KTO if your feedback is thumbs-up/thumbs-down rather than pairs.

DPO vs KTO vs ORPO at a Glance

Results are each paper's own claims, from the abstracts. They were measured on different models and benchmarks, so they are not a head-to-head ranking.

What changed since these papers (SimPO and GRPO)

• SimPO (Meng et al., May 2024) drops the reference model and uses the length-normalized log-probability as the reward. Its authors report it beats DPO by up to 6.4 points on AlpacaEval 2 and 7.5 on Arena-Hard. TRL's DPO trainer exposes SimPO-style length normalization as loss_type="sigmoid_norm".
• GRPO (DeepSeekMath, February 2024) is online reinforcement learning without a critic model: it samples a group of answers per prompt and scores each against the group. DeepSeek used it to train DeepSeek-R1, and it is now the usual choice when answers can be checked automatically (maths, code, tool calls). TRL ships it as GRPOTrainer.
• Rule of thumb: offline preference data about style, tone or helpfulness → DPO, ORPO or KTO. Verifiable right/wrong answers and a reward function → GRPO.

By 2026 all three are standard tools in local fine-tuning pipelines. They share a common goal — align the model to your preferences — but use different mathematical formulations and dataset assumptions. Updated November 2, 2026: decision table with paper-cited results, TRL code for the current API, and the SimPO/GRPO note above.

This guide covers everything: the math behind each method (skipping RLHF's reward-model + PPO complexity), dataset formats, when to use which, integration with QLoRA via TRL and Axolotl, hyperparameter tuning, common failure modes, and concrete recipes for aligning Llama 3.1 8B to a specific brand voice or domain style.

Table of Contents

1. DPO vs KTO vs ORPO at a Glance
2. Why Preference Fine-Tuning Matters
3. The RLHF Pipeline (and Why It Was Replaced)
4. DPO: Direct Preference Optimization
5. ORPO: Unified SFT + Preference
6. KTO: Unpaired Preference Signals
7. Choosing DPO vs ORPO vs KTO
8. Dataset Format and Sources
9. TRL Integration
10. Axolotl Integration
11. Hyperparameter Tuning
12. QLoRA + DPO/ORPO/KTO
13. Evaluating Preference-Tuned Models
14. Alignment Tax and Mitigations
15. Common Failures
16. FAQ

---

Why Preference Fine-Tuning Matters

Standard SFT (supervised fine-tuning) teaches a model to imitate target outputs. But often you have multiple acceptable responses with quality differences — and you want the model to learn which is better, not just which appeared in your training set.

Preference fine-tuning operationalizes "better than": train the model to assign higher probability to preferred responses than rejected ones.

Use cases:

• Style adaptation (brand voice, formality, humor)
• Helpfulness vs harmlessness trade-offs
• Refusal behavior tuning
• Domain-specific quality (medical accuracy, legal citation style)
• A/B test winning patterns

---

The RLHF Pipeline (and Why It Was Replaced)

Classical RLHF (used by InstructGPT, ChatGPT, early Llama 2 chat):

Step 1: SFT on demonstrations
Step 2: Train a reward model on preference pairs
Step 3: Use PPO reinforcement learning to optimize policy against reward model

Problems: PPO needs the policy, a reference model, a reward model and a value model in memory at once, samples from the model during training, is sensitive to hyperparameters, and is prone to reward hacking.

DPO showed that the reward model and PPO step can be eliminated — directly optimize the policy on preferences via a closed-form loss. Offline preference tuning of open-weight models is now mostly DPO and its variants; online RL came back for reasoning in the form of GRPO (see the note at the top).

---

DPO: Direct Preference Optimization

The DPO loss for a preference pair (prompt x, chosen y_w, rejected y_l):

L_DPO = -log σ(β · log π(y_w|x)/π_ref(y_w|x) - β · log π(y_l|x)/π_ref(y_l|x))

Where π is the model being trained, π_ref is a frozen reference (typically the SFT checkpoint), and β controls deviation magnitude.

Intuition: increase the log-probability of chosen responses relative to rejected, weighted against the reference model. β=0.1 is the typical default.

Pipeline:

1. SFT on demonstrations (gives π_ref)
2. DPO on preference pairs (yields aligned π)

---

ORPO: Unified SFT + Preference

ORPO (Hong et al., March 2024) unifies SFT and DPO into a single loss:

L_ORPO = L_SFT(y_w) + λ · L_OR(y_w, y_l)

Where L_SFT is standard SFT loss on chosen responses, and L_OR is an odds-ratio loss penalizing rejected responses. λ controls the strength of preference signal.

Result: skip the SFT-then-DPO two-stage pipeline and the reference model. Train once, get both. The paper's headline: Phi-2, Llama-2 7B and Mistral 7B trained this way on UltraFeedback alone reach up to 12.20% on AlpacaEval 2.0 and 7.32 on MT-Bench. In TRL the λ weight is the beta argument of ORPOConfig (default 0.1).

Use ORPO when starting from a base (non-SFT) model. Use DPO when you already have an SFT checkpoint.

---

KTO: Unpaired Preference Signals

KTO (Ethayarajh et al., 2024) handles binary feedback without paired comparisons:

L_KTO = E[w_y · σ(β · (log π(y|x)/π_ref(y|x) - z_ref))]

Where w_y is +1 for "good" responses, -1 for "bad", and z_ref is a baseline. Loss derived from Kahneman-Tversky prospect theory (loss-aversion modeling).

Useful when:

• You have user thumbs-up/thumbs-down without paired comparisons
• A/B test outcomes (winner / loser of single response shown)
• Automated quality classifier scores
• Larger datasets of unpaired feedback than paired

The KTO paper reports it matches or exceeds preference-based methods at scales from 1B to 30B. TRL can also convert a paired dataset into KTO format automatically (chosen → desirable, rejected → undesirable).

---

Choosing DPO vs ORPO vs KTO

---

Dataset Format and Sources

DPO / ORPO format:

{
  "prompt": "What is local AI?",
  "chosen": "Local AI is...",
  "rejected": "Local AI is some thing..."
}

KTO format (TRL field names):

{"prompt": "What is local AI?", "completion": "Local AI is...", "label": true}
{"prompt": "What is local AI?", "completion": "Local AI is some thing...", "label": false}

Both formats also accept conversational messages (lists of role/content dicts); TRL applies the chat template for you.

Sources:

• HH-RLHF (Anthropic) — helpfulness/harmlessness pairs
• UltraFeedback — broad preference pairs
• OpenAssistant — community-rated responses
• Synthetic AI judge: generate 2 responses, ask GPT-4o/Claude which is better
• Production logs with thumbs-up/down (KTO)
• Manual annotation for brand voice / specific tone

For most domain adaptation, AI-judge synthetic pairs are the cheapest practical route; spot-check a sample by hand before training.

---

TRL Integration

Current TRL (checked against the main-branch docs, November 2, 2026). The tokenizer is passed as processing_class (the old tokenizer= argument is gone), and ref_model is optional — if you omit it, TRL uses the starting model as the reference.

from datasets import load_dataset
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",          # or your SFT checkpoint
    args=DPOConfig(
        output_dir="./dpo_output",
        beta=0.1,
        learning_rate=1e-5,           # TRL's tip for LoRA adapters; ~1e-6 for full fine-tuning
        num_train_epochs=1,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
    ),
    train_dataset=dataset,
    peft_config=LoraConfig(),         # drop this line for full fine-tuning
)
trainer.train()

ORPO now lives in TRL's experimental module:

from trl.experimental.orpo import ORPOConfig, ORPOTrainer

trainer = ORPOTrainer(model=model, args=ORPOConfig(output_dir="./orpo_output", beta=0.1),
                      processing_class=tokenizer, train_dataset=dataset)

KTO uses the unpaired format and its own config:

from trl import KTOConfig, KTOTrainer

trainer = KTOTrainer(model="Qwen/Qwen3-0.6B",
                     args=KTOConfig(output_dir="./kto_output", per_device_train_batch_size=4),
                     train_dataset=load_dataset("trl-lib/kto-mix-14k", split="train"))

---

Axolotl Integration

base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
adapter: qlora
load_in_4bit: true
lora_r: 32
lora_alpha: 64

rl: dpo                       # or orpo, kto
dpo_beta: 0.1
datasets:
  - path: ./preferences.jsonl
    type: chatml.intel
    split: train

learning_rate: 5e-7
num_epochs: 1
micro_batch_size: 2
gradient_accumulation_steps: 4

accelerate launch -m axolotl.cli.train dpo.yml

---

Hyperparameter Tuning

Defaults and ranges from the TRL documentation for each trainer. For KTO with imbalanced data, set desirable_weight / undesirable_weight so the weighted ratio of positives to negatives lands between 1:1 and 4:3.

Lower learning rates than SFT! Preference fine-tuning is sensitive — too high LR causes catastrophic forgetting.

---

QLoRA + DPO/ORPO/KTO

Combining 4-bit quantized base with preference fine-tuning:

# Axolotl
adapter: qlora
load_in_4bit: true
lora_r: 32
lora_alpha: 64
rl: dpo
dpo_beta: 0.1

Memory: similar to QLoRA SFT. Time: longer than SFT on the same data, because DPO scores both the chosen and the rejected response and also needs reference log-probabilities (TRL can precompute them with precompute_ref_log_probs=True to save memory). With LoRA, TRL uses the base model with the adapter switched off as the reference, so no second copy is loaded.

How long a run takes depends on model size, number of pairs, sequence length and GPU; run 50 steps first and extrapolate.

---

Evaluating Preference-Tuned Models

Three evaluation axes:

1. Held-out preference accuracy: does the model prefer chosen over rejected on held-out pairs?
2. General benchmark regression: re-run MMLU / HumanEval / GSM8K and compare against the starting model; any clear drop is the alignment tax
3. Qualitative: spot-check 50-100 outputs by hand

For brand voice / style adaptation: human evaluation matters more than benchmarks. Have target users rate 30-50 outputs blind vs the SFT-only model.

---

Alignment Tax and Mitigations

Preference fine-tuning often degrades unrelated capabilities — the "alignment tax." Mitigations:

1. Higher β (0.3-0.5) — limits deviation from reference
2. Fewer epochs (1-2 max)
3. Mix in SFT data — 30-50% general SFT examples in the training mix
4. Smaller LoRA rank — limits capacity for over-specialization
5. Regular evaluation on out-of-domain benchmarks during training

---

Common Failures

---

FAQ

See answers to common DPO/ORPO/KTO questions below.

---

Sources: DPO paper (Rafailov et al., 2023) | ORPO paper (Hong et al., 2024) | KTO paper (Ethayarajh et al., 2024) | SimPO paper (Meng et al., 2024) | DeepSeekMath / GRPO (Shao et al., 2024) | TRL DPO, ORPO and KTO docs | HuggingFace TRL | Axolotl.

Related guides:

• LoRA and QLoRA Fine-Tuning Local Guide
• Train AI Model Own Data Locally
• Build Your First AI Dataset
• Synthetic vs Real Data