Which GGUF quantization level should you pick?

GGUF is the file format llama.cpp and most of its ecosystem, including Ollama and LM Studio, use to package a quantized model plus its metadata in one file. Open a GGUF repo on Hugging Face and you will see a list of files named things like Q4_K_M, Q5_K_S, and Q8_0: the same model, quantized to different precisions. Pick carelessly and you either waste memory you did not need to spend, or load a version too degraded to trust. This guide has the real numbers behind that choice, pulled from the tools that actually produce these files.

What quantization actually does

A freshly trained model stores each parameter (weight) as a 16-bit or 32-bit floating-point number. Quantization replaces that with fewer bits per weight, commonly 8, 6, 5, 4, 3, or 2. Fewer bits per weight means less precision for each number, but also less data to store overall, so the whole file shrinks. An 8-billion-parameter model at full 16-bit precision runs close to 15 GB just for the weights. Quantized to Q4_K_M, the same model needs under 5 GB.²

Model weights are not equally important, and most cluster in a narrow range around zero. That is why trimming precision does not break a model outright: it degrades gradually, at least down to a point, and that gradual curve is the entire reason quantization is worth doing. The numbers below show exactly how gradual, and where it stops being gradual.

That is the whole trade: fewer bits per weight buys less VRAM and RAM, a smaller download, and usually faster inference, in exchange for some accuracy. The rest of this guide is about where that trade stops being worth it, for your hardware and your use case.

Decoding the GGUF name

Take Q4_K_M apart and it reads as three pieces. Q4 is 4 bits per weight, on average. K means it uses the newer "k-quant" method, covered below. S, M, or L (Small, Medium, Large) says how much of the model gets the full bit count versus a lower one: an _M file spends a little more precision on certain tensors than _S does, and a little less than _L.¹ A file named just Q4_0, with no K, is the older legacy method described further down.

The bit count in the name is an average, not an exact figure. K-quants group weights into "super-blocks" that share a scale, and different tensors inside the model, attention weights versus feed-forward weights for example, can be assigned a different quant type within the same file. That is why Q4_K_M measures out to roughly 4.5 to 4.9 bits per weight in practice, not a flat 4.¹

The quant-level comparison table

These sizes and bit counts are real measurements against the Llama-3.1-8B model, taken from the tool that produces GGUF quantizations.² The quality-cost column comes from a different, older benchmark: the original k-quants pull request measured perplexity on a 7B LLaMA model against the wikitext dataset, where full 16-bit precision (F16) scored 5.9066.³ The two were not measured on the same model, so read the percentages as directional. Perplexity shift depends on the architecture and the fine-tune, and yours will differ somewhat.

What perplexity means, in plain terms

Perplexity measures how surprised a model is by real text it was not trained to memorize.⁴ Feed it a held-out passage and, at each word, check how confidently it predicted what came next, then average that across the passage into one number. Lower is better: a perplexity of roughly 6 means the model was choosing among about 6 equally likely next words on average, while a perplexity of 7 means it was less certain, spreading its guess across more options. It is not a percentage or a pass/fail score, and it does not measure any one skill like coding or math directly. It is a rough, model-wide gauge of fluency loss: useful for comparing one quantization of a model against itself, less useful for comparing across different models or architectures.

K-quants versus older legacy quants

Before k-quants, llama.cpp used a simpler scheme: Q4_0, Q5_0, Q8_0, and their variants, which round each weight to the nearest representable value within a small block of 32 weights, using one scale per block.¹ It works, but it spends the same precision everywhere, whether a given weight matters a lot to the output or barely at all.

• K-quants (the _K suffix) group weights into larger super-blocks and let different parts of the model use different precision, spending more bits where it helps and fewer where it does not.¹
• At the same bit budget, a K-quant keeps noticeably closer to full precision than the legacy type it replaces, which is why the comparison table above uses K-quants throughout.
• The legacy Q4_0/Q5_0/Q8_0 family still shows up mainly for backward compatibility and specific hardware paths; Hugging Face's own documentation now labels them "not used widely as of today."¹

Importance matrix (imatrix) and I-quants, briefly

An importance matrix is calibration data, real text run through the full-precision model before quantizing, that records which weights actually move the output on typical input. Quantizing with an imatrix (the --imatrix flag in llama.cpp's quantize tool) spends the bit budget more carefully: weights the matrix flags as important keep more precision, and the rest give up a little more.⁵ It is optional for most K-quants and close to essential for the smallest ones.

I-quants (IQ4_XS, IQ3_S, IQ2_XXS, and similar) are a further family built to use an imatrix from the start, reaching down to roughly 2 bits per weight and below while staying more coherent than a legacy quant at the same size.¹ They take longer to produce and are not always as fast to run as a K-quant, so reach for them only once you are already below Q3 and need every bit of quality you can get.

Estimating VRAM before you download anything

The weights alone are roughly params times bits per weight, divided by 8 for bytes. An 8-billion-parameter model at 4.89 bits/weight (Q4_K_M) works out to about 8,000,000,000 × 4.89 ÷ 8 ≈ 4.9 GB, which matches the measured 4.58 GB closely enough for planning.² That covers only the weights, though.

• Weights: params × bits/weight ÷ 8. This is the floor. It has to fit in VRAM (or RAM, for CPU inference) before anything else does.
• KV cache: memory that grows with context length and the number of layers and attention heads. A long conversation or a large pasted file can add real gigabytes on top of the weights.
• Runtime overhead: the inference engine, buffers, and anything else sharing the GPU. Budget headroom for this; it is not a number you can shave to zero.

As a starting rule of thumb, leave 10 to 20% of your VRAM free above the weight size for short conversations, and more if you plan to run a large context window regularly. If a quant just barely fits the raw weights, it will not fit once you start actually using it.

Choosing by your hardware

These are starting points built from the same weight-size math above, not a promise for any specific model. Always check the exact file size on the model page before committing to a multi-gigabyte download.

Bigger model at a lower quant, or smaller model at a higher quant?

Given a choice between a bigger model quantized lower and a smaller model quantized higher, at roughly the same file size, prefer the bigger model, down to about Q4. The pull request that introduced k-quants made exactly this case with data: it plotted perplexity against file size for LLaMA 7B, 13B, 30B, and 65B, each at several quant levels, and the larger models at lower precision consistently beat the smaller models at higher precision for the same size budget.³ A bigger model has more to say even at a slightly blurred precision than a smaller model says clearly.

Where GGUF files actually come from

Almost nobody quantizes their own models from scratch. Most GGUF files on Hugging Face's GGUF listing were converted by the model's original publisher or by a community contributor, using llama.cpp's own conversion script to go from the original format, often safetensors, into GGUF, and then the llama-quantize tool to produce each Q-level.² Hugging Face also hosts a no-setup conversion tool that runs the same pipeline for you, for the rare case where a model has not been published in GGUF yet.¹

When you are choosing a download, check the model card for which quants the publisher actually tested, and prefer a repo that used an imatrix if you are going below Q4. Not every uploader calibrates with one, and the ones that do usually say so.

How to choose, in short

• Default to Q4_K_M. It is the standard recommendation for a reason: under 1% perplexity cost for a real cut in memory.²
• Go up to Q5_K_M or Q6_K if the model fits with room to spare and you want the last bit of quality for close to free.
• Go down to Q3 or lower only when it is the difference between running a model at all and not running it, and prefer an imatrix or I-quant version if you do.
• Given equal file size, pick the bigger model at a lower quant over a smaller model at a higher one, as long as you are staying at Q4 or above.
• Always budget beyond the weight size for the KV cache and runtime overhead before assuming a quant will fit on your GPU.