LLM Quantization Explained — VRAM Savings vs Quality Trade-offs
Quantization reduces model weights from 16-bit floats to 8-bit, 4-bit, or lower integers. This cuts VRAM requirements by 50–75% with modest quality loss. This guide covers every quantization level and when to use each.
What quantization does
LLM weights are stored as floating-point numbers. FP16 uses 2 bytes per parameter. Q4_K_M uses ~0.5 bytes — a 4× reduction. A 70B parameter model at FP16 needs 140GB VRAM; at Q4_K_M, ~43GB. Quantization maps continuous float values to a discrete set of integers, introducing rounding error (quality loss).
Quantization levels compared
FP16: 2 bytes/param, full quality. Q8_0: 1 byte, near-lossless. Q6_K: 0.75 bytes, very good. Q5_K_M: 0.625 bytes, good. Q4_K_M: 0.5 bytes, good for most tasks. Q3_K_M: 0.375 bytes, noticeable degradation on complex tasks. Q2: ~0.25 bytes, significant quality loss — avoid for production.
K-quants vs older quantization
The _K suffix (e.g. Q4_K_M) refers to k-quants — a smarter quantization that uses mixed precision (some layers at Q6, others at Q4) to minimize quality loss at the same average bit depth. _K_M is the recommended middle-of-the-road version; _K_S saves ~5% more VRAM at slight quality cost.
When quantization is appropriate
Q4_K_M is appropriate for: classification, summarization, Q&A, code completion. Q5_K_M or Q6_K for: precise reasoning, legal/financial analysis, tasks where errors are costly. FP16 for: fine-tuning, benchmarking, research where reproducibility matters.
Related calculators
Related guides
Frequently Asked Questions
Does quantization affect generation quality?
At Q4_K_M: typically 1-3% benchmark degradation vs FP16 on standard tasks. For complex multi-step reasoning, the gap widens to 5-8%. For simple extraction and classification, Q4 is often indistinguishable from FP16.
Which quantization should I use for a 70B model on consumer hardware?
Q4_K_M is the standard recommendation for consumer hardware: good quality, 43GB VRAM for 70B (two RTX 4090s). If you have an RTX 5090 32GB: Q5_K_M gets you the whole 34B Scout model at good quality solo.
Guide updated 2026-08-20. Pricing data verified weekly. See methodology · sources.