Fine-Tuning LLMs · Lesson 6
Reasoning & Style Alignment
After SFT, preference tuning aligns outputs with what people prefer. RLHF trains a reward model from human rankings and optimises the policy with reinforcement learning (PPO).
DPO (Direct Preference Optimization) skips the separate reward model and learns directly from chosen/rejected pairs; ORPO and KTO are related, simpler variants.
For reasoning, models can be trained on step-by-step solutions and with reinforcement learning on verifiable rewards — such as correct maths answers or passing tests — as in GRPO.
Key points
- RLHF: reward model + RL
- DPO learns directly from preference pairs
- Align tone, format and helpfulness
- RL with verifiable rewards for reasoning