The short answer: start at Q4_K_M. It is the tier that keeps most of a model’s behaviour while cutting the 16-bit size to roughly a quarter, and on consumer cards it is the only tier that usually makes a large model fit at all. Move up to Q6_K or Q8_0 when memory is not the constraint and the task is sensitive; move down to Q3_K_M only when Q4_K_M will not fit; treat Q2_K as a last resort for draft or retrieval work rather than for conversation.
The rest of this page is the comparison behind that answer: what each tier stores per parameter, what it costs in file size at three model scales, and what quality you give up at each step.
Tier by tier: what each one stores per parameter
The single number that drives memory is bits per weight. Lower it and the file shrinks in direct proportion. The values below are widely used approximations for GGUF tiers, not exact figures for every model.
| Tier | Approx bits/param | Compared with the tier above | What it is for |
|---|---|---|---|
| F16 | 16 | Baseline: unquantized 16-bit reference | The size every other row is measured against |
| Q8_0 | 8.5 | Roughly half of F16 | Near-lossless; pick it when memory is not the limit |
| Q6_K | 6.6 | About 22% smaller than Q8_0 | Near-original behaviour at a real saving |
| Q5_K_M | 5.5 | About 17% smaller than Q6_K | A modest quality step above the default |
| Q4_K_M | 4.5 | About 18% smaller than Q5_K_M | The common default: the knee of the curve |
| Q3_K_M | 3.5 | About 22% smaller than Q4_K_M | Fits where Q4 will not; behaviour starts to show it |
| Q2_K | 2.6 | About 26% smaller than Q3_K_M | Aggressive compression, smallest files |
These are averages. A tier like Q4_K_M stores some layers at more than 4.5 bits and others less, which is why the per-parameter figure is a blend rather than a constant. The saving column is arithmetic on the two bits figures; the real file differs by a few percent, as the last section explains.
What each tier costs in file size at 7B, 13B and 70B
Apply weight size = parameters × bits ÷ 8 (with parameters in billions). The table converts each tier to real file sizes.
| Tier | Bits | 7B weight | 13B weight | 70B weight |
|---|---|---|---|---|
| Q2_K | 2.6 | 2.28 GB | 4.23 GB | 22.8 GB |
| Q3_K_M | 3.5 | 3.06 GB | 5.69 GB | 30.6 GB |
| Q4_K_M | 4.5 | 3.94 GB | 7.31 GB | 39.4 GB |
| Q5_K_M | 5.5 | 4.81 GB | 8.94 GB | 48.1 GB |
| Q6_K | 6.6 | 5.78 GB | 10.7 GB | 57.8 GB |
| Q8_0 | 8.5 | 7.44 GB | 13.8 GB | 74.4 GB |
| F16 | 16 | 14.0 GB | 26.0 GB | 140 GB |
Read the table by column to see why the tier matters more than the model scale. A 70B model at F16 needs 140 GB, which no consumer card offers. At Q4_K_M it drops to about 39 GB, which is reachable across two 24 GB cards or one large unified-memory machine. The 7B column, by contrast, fits at any tier on a 16 GB card — so for small models the choice is about quality headroom, not about fitting. The same numbers appear per model in the file size lookup.
The tradeoff table: memory saved versus quality lost
The memory saving is exact and linear. The quality cost is not a single number and depends on the model and the task, so the right-hand column states a direction and a rough size, not a score. The direction is consistent: lower bits tend to hurt instruction following and multi-step reasoning before they move perplexity much.
| Move | Memory effect | Typical quality effect |
|---|---|---|
| F16 → Q8_0 | Save ~half | Usually imperceptible |
| Q8_0 → Q6_K | Save ~22% | Small, often unnoticeable |
| Q6_K → Q5_K_M | Save ~17% | Minor on most tasks |
| Q5_K_M → Q4_K_M | Save ~18% | Noticeable only on hard prompts |
| Q4_K_M → Q3_K_M | Save ~22% | Clear on reasoning and format |
| Q3_K_M → Q2_K | Save ~26% | Rough, best avoided for chat |
The pattern to internalize: each step down saves a comparable chunk of memory, but the quality penalty grows faster at the bottom of the table. Q4_K_M is the knee of the curve for most people, which is why it became the default download. What is actually being lost at each step, and how confident anyone can be about it, is the subject of quantization quality loss.
How to pick: a decision checklist
Work down the list and stop at the first row that applies.
| Your situation | Pick | Why |
|---|---|---|
| The model fits at Q4_K_M with a gigabyte or more to spare | Q4_K_M | The default; further compression buys nothing you need |
| It fits, and the task is sensitive (long reasoning, strict formats) | Q6_K, or Q8_0 with plenty of headroom | You are spending memory you already have on behaviour |
| It does not fit at Q4_K_M, and a smaller model is not acceptable | Q3_K_M | The next step that usually still behaves; expect format slips |
| Nothing above fits, and the use is draft or retrieval | Q2_K | Usable for short, low-stakes generation only |
| A 7B fits at any tier | Q8_0 or F16 | Small models have the headroom; use it |
| Even Q2_K does not fit | A smaller model | Below Q2_K there is little left to give; change the model, not the tier |
Below Q2_K there is little point; you have left too much precision behind. A 13B at Q4_K_M is a different tool than a 70B at Q2_K, and the smaller model will usually behave better despite fewer parameters. Match the tier to the task, not just to the memory limit.
The VRAM calculator takes a bits-per-parameter value directly, so you can test a tier against your card before downloading. The 70B local rundown shows which tiers are realistic on common hardware, and mixture-of-experts memory covers the models where the active-parameter count does not describe what has to be resident.
A note on real file sizes
The sizes here come from the formula and the approximate bits above. Your actual GGUF may differ by a few percent because embedding layers, the output tensor, and per-layer block choices are not uniform. Always read the real file size from the download page or the file itself; treat this table as the plan, not the receipt.
When quantization is not enough
Below Q2_K there is little point; you have left too much precision behind. At that point the right move is a smaller model, not a lower tier. A 13B at Q4_K_M is a different tool than a 70B at Q2_K, and the smaller model will usually behave better despite fewer parameters. Match the tier to the task, not just to the memory limit.