Fine-Tuning LLMs · Lesson 4
Domain-Adaptive Pre-Training (DAPT)
DAPT continues the model's original next-token pre-training on large amounts of unlabelled domain text — medical papers, legal filings, financial reports.
It teaches vocabulary, style and knowledge of the domain. It is usually followed by SFT so the model also learns to follow instructions.
It needs much more data and compute than SFT, and can cause catastrophic forgetting of general skills, so mix in some general data.
Key points
- Continued pre-training on raw domain text
- Learns domain language and knowledge
- Usually followed by SFT
- Mix general data to limit forgetting