Quantization explained: GGUF Q4 vs Q5 vs Q8

Quantization is a deliberate quality/size trade. Local LLM users meet it as Q4_K_M, Q5_K_M, Q8_0 file names.

What changes

Lower-bit quants store weights with less precision, shrinking download size and VRAM. Higher-bit quants keep more fidelity and cost memory.

Different models degrade differently. A coding model and a chat model will not show the same Q4 vs Q8 gap on your prompts.

How VRAMwise treats sizes

When our model table has published GGUF sizes per quant, the fit checker uses those numbers. When you only know parameter count, the quant compare tool uses transparent planning stand-ins labeled as such.

Practical order

• Check fit at Q4 with your target context.
• If headroom is large, try Q5 on the same prompts.
• Use Q8 when quality regressions are visible and VRAM allows.

General information, not personalized advice.