GGUF Quantization Compared: What Q4_K_M Actually Costs You
GGUF quant tiers Q2_K to F16: bits per parameter and the 7B, 13B, and 70B weight sizes, plus the quality-versus-memory tradeoff.
GGUF Quantization Compared: What Q4_K_M Actually Costs You
Quantization is the dial that decides whether a model fits your card at all. This page lists the common GGUF tiers, the bits per parameter each stores, the resulting weight sizes for three model scales, and the quality tradeoff you accept when you drop a tier.
Bits per parameter by tier
The single number that drives memory is bits per weight. Lower it and the file shrinks in direct proportion. The values below are widely used approximations for GGUF tiers, not exact figures for every model.
| Tier | Approx bits/param | What it is |
|---|---|---|
| Q2_K | 2.6 | Aggressive compression, smallest files |
| Q3_K_M | 3.5 | Low-bit, fits where Q4 will not |
| Q4_K_M | 4.5 | The common default tradeoff |
| Q5_K_M | 5.5 | Higher quality, modest size bump |
| Q6_K | 6.6 | Near-original weights |
| Q8_0 | 8.5 | Near-lossless, still compressed |
| F16 | 16 | Unquantized 16-bit reference |
These are averages. A tier like Q4_K_M stores some layers at more than 4.5 bits and others less, which is why the per-parameter figure is a blend rather than a constant.
Weight size at three model scales
Apply weight size = parameters × bits ÷ 8 (with parameters in billions). The table converts each tier to real file sizes.
| Tier | Bits | 7B weight | 13B weight | 70B weight |
|---|---|---|---|---|
| Q2_K | 2.6 | 2.28 GB | 4.23 GB | 22.8 GB |
| Q3_K_M | 3.5 | 3.06 GB | 5.69 GB | 30.6 GB |
| Q4_K_M | 4.5 | 3.94 GB | 7.31 GB | 39.4 GB |
| Q5_K_M | 5.5 | 4.81 GB | 8.94 GB | 48.1 GB |
| Q6_K | 6.6 | 5.78 GB | 10.7 GB | 57.8 GB |
| Q8_0 | 8.5 | 7.44 GB | 13.8 GB | 74.4 GB |
| F16 | 16 | 14.0 GB | 26.0 GB | 140 GB |
A 70B model at F16 needs 140 GB, which no consumer card offers. At Q4_K_M it drops to about 39 GB, which is reachable across two 24 GB cards or one large unified-memory machine. That single column is why quantization is the gatekeeper for local large models.
Quality versus memory saved
The memory saving is exact and linear. The quality cost is not a single number and depends on the model and the task. The direction is consistent: lower bits save more memory and tend to hurt instruction following and multi-step reasoning before they move perplexity much.
| Move | Memory effect | Typical quality effect |
|---|---|---|
| F16 → Q8_0 | Save ~half | Usually imperceptible |
| Q8_0 → Q6_K | Save ~22% | Small, often unnoticeable |
| Q6_K → Q5_K_M | Save ~17% | Minor on most tasks |
| Q5_K_M → Q4_K_M | Save ~18% | Noticeable only on hard prompts |
| Q4_K_M → Q3_K_M | Save ~22% | Clear on reasoning and format |
| Q3_K_M → Q2_K | Save ~26% | Rough, best avoided for chat |
The pattern to internalize: each step down saves a comparable chunk of memory, but the quality penalty grows faster at the bottom of the table. Q4_K_M is the knee of the curve for most people, which is why it became the default download.
How to pick a tier in practice
- Start at Q4_K_M. If the model fits with headroom, you are done.
- If it does not fit, drop to Q3_K_M before you consider a smaller model; the behavioral hit is usually acceptable for draft or retrieval use.
- If you have memory to spare, move up to Q6_K or Q8_0. The gain is real but small unless the task is sensitive.
- Avoid Q2_K for anything you will actually chat with; the compression is too aggressive to keep coherence.
The VRAM calculator takes a bits-per-parameter value directly, so you can test a tier against your card before downloading. The 70B local rundown shows which tiers are realistic on common hardware.
A note on real file sizes
The sizes here come from the formula and the approximate bits above. Your actual GGUF may differ by a few percent because embedding layers, the output tensor, and per-layer block choices are not uniform. Always read the real file size from the download page or the file itself; treat this table as the plan, not the receipt.
When quantization is not enough
Below Q2_K there is little point; you have left too much precision behind. At that point the right move is a smaller model, not a lower tier. A 13B at Q4_K_M is a different tool than a 70B at Q2_K, and the smaller model will usually behave better despite fewer parameters. Match the tier to the task, not just to the memory limit.
Frequently asked questions
What does Q4_K_M mean in plain terms?
It is a GGUF quant tier storing roughly 4.5 bits per weight on average. The _K_M variant mixes block sizes so medium layers keep slightly more precision. Treat 4.5 as an approximation; the real file size varies a few percent by model.
Is lower always worse quality?
Lower bits save memory but tend to hurt instruction following and long-chain reasoning before they hurt raw perplexity. The damage shows up in behavior, not just benchmark numbers, so the tradeoff is real even when scores look close.
Why is F16 listed at 16 bits and not 2?
F16 is the unquantized 16-bit floating point format. It is the reference size; everything below it is a compression of those weights, and the bits-per-parameter column shows how aggressive each compression is.
Can I trust the table sizes exactly?
Use them as planning estimates. Different models have different layer and embedding counts, so the same tier yields file sizes that differ by a few percent. Always confirm against the actual downloaded file.
Which tier is the usual sweet spot?
Q4_K_M is the common default for fitting large models on consumer cards because it keeps most of the quality while roughly quartering the 16-bit size. Q8_0 is the choice when you have headroom and want near-lossless weights.