Fine-Tuning LLMs · Lesson 8
Quantization & Deployment
Quantisation stores weights with fewer bits — 8-bit or 4-bit instead of 16-bit — cutting memory and often speeding inference, with a small accuracy loss.
Common formats: GPTQ and AWQ for GPU serving, GGUF for llama.cpp and Ollama on CPUs and laptops, and bitsandbytes for on-the-fly 4/8-bit loading.
Merge LoRA adapters into the base model or serve them separately with vLLM or TGI, then benchmark quality, latency and throughput before release.
Key points
- Fewer bits → less memory, faster inference
- GPTQ/AWQ for GPUs, GGUF for local CPUs
- Merge or hot-swap LoRA adapters
- Benchmark quality and speed after quantising