Quantization Quality Loss: What Is Known, What Is Assumed

A source-graded breakdown of what quantization does to model behavior: directional claims on quality loss, evidence strength, and how to verify it yourself.

By VRAM Fit editorial Published 2026-09-17 Updated 2026-09-17

Quantization quality loss is the part of the memory tradeoff that no bits-per-parameter table can show. This page does not assign a score to any tier. It lists what can be said about quality loss, and grades each claim by the strength of its evidence: a public benchmark, a vendor statement, broad community experience, or theory alone. The point is to separate what is measured from what is assumed.

How to read the evidence grades

Every claim below carries one of four grades. The grade is the real content of this page; the words attached to it are only meaningful next to the grade.

Grade	Meaning	What it lets you conclude
A — Public benchmark support	Reproducible results published across more than one model or task	A broad direction holds; do not read it as an exact per-tier number
B — Vendor documentation	Stated by the engine or format authors	The format is designed that way; behavioral outcome is described, not scored
C — Broad community experience	Repeatedly reported by users, not from controlled studies	Likely true in practice; sensitive to model, prompt, and settings
D — Theoretical inference only	Follows from how quantization math works, not from measurement	Plausible and expected; unverified for your specific case

A grade is not a quality score. Grade A does not mean a tier is good; it means a statement about it is well supported.

The claims, graded

The table collects the directional statements that are safe to make about quantization and quality. None of them is a number. Where a number would usually appear, it is left out on purpose.

Claim	Grade	Why this grade
Lower bits raise perplexity monotonically in the aggregate	A	Perplexity is a wide-aperture, low-variance metric; published results across models show the trend consistently
Low-bit damage to instruction following appears before damage to perplexity	C	Widely reported across chat and agent use; some public benchmarks touch it but task dependence is high
Low-bit damage to long-chain reasoning appears before damage to perplexity	C	Repeatedly observed on multi-step and tool-use tasks; not captured well by short-form evals
The effect is stronger on small models than on large ones	C	Community reports and the redundancy argument both point this way; exact cross-over varies
K-quant _M variants are gentler than older formats at the same average bits	B	Stated by the format authors; the mixed block sizes are a documented design choice
Below roughly Q3, open chat coherence drops sharply	C	Broad user experience; some models hold up, others do not, at the same tier
Faithful formatting, code, and math degrade earlier than free-form prose	C	Reported on structured-output and coding tasks; tied to low-bit sensitivity of exact tokens
Quantization can shift bias and safety alignment in unpredictable directions	D	Follows from weight perturbation theory; measured direction is not predictable from the tier alone
Two engines at the same nominal tier can differ slightly in behavior	C	Different block choices and implementations are reported to vary on edge cases

The pattern that matters: a single bit count hides two different stories. One story is the average loss measured by a smooth metric; the other is the abrupt, task-specific loss that shows up only when the model has to follow a rule or chain ten steps. The second story is graded C because it is real but not reduced to a figure.

Why quality loss resists a one-line number

Three structural reasons explain why no page, including this one, should print a “percent quality lost at Q4_K_M.”

Task sensitivity differs. A tier that keeps free chat readable can still break JSON output or a multi-hop reasoning chain. The same weights score differently on a trivia set and on a function-calling suite, so any single number averages away the part you care about.

Eval sets carry bias. Public benchmarks are written by someone, for some models, in some language and style. A tier that looks fine on one suite can stumble on prompts drawn from a different domain. The benchmark grade above supports a direction, not a transferable score.

Sampling settings move the result. Temperature, top-p, seed, and even the exact decode path change which token the model picks when several are close. Quantization shrinks the gap between near-ties, so the same tier can look worse or better depending on how it is run. That is why a self-check (below) beats a quoted score.

These three reasons are themselves graded C to D. They are expectations about measurement, not measurements.

Directions that are safe to state

From the graded table, three directions can be stated without a number attached:

Low-bit compression harms instruction following and long reasoning earlier than it harms perplexity. The damage shows up in behavior before it shows up in a smooth metric.
The same tier hurts a small model more than a large one, because there is less redundancy for the error to spread across.
The _M variants of K-quant keep more behavior at the same average bit width than the older flat formats, because medium layers retain extra precision.

None of these says how much. They say where to look, and in what order the loss tends to arrive.

How to run a minimal self-check

A reader who wants to know the real cost of a tier on their own workload can run a small, reproducible comparison. The method is graded C as practical guidance, not as a measured claim.

Choose a fixed prompt set of 20 to 50 items that mirror the real task: the same formatting, the same domain, the same length.
Run the same model at two tiers, for example Q4_K_M and Q6_K, on the same engine and hardware.
Fix temperature and seed where the engine allows, so differences come from the weights and not from sampling.
Record the outputs side by side and mark only the differences you can reproduce on a second run.
Judge on the behavior that matters to you: does the format hold, does the chain complete, does the answer stay correct.

The result is a personal, task-specific reading of quality loss. It will not match a leaderboard, and it should not need to. For the tier math behind the comparison, the quantization comparison page lists bits per parameter, and the VRAM calculator turns a tier into a memory number you can plan against.

Where the loss is independent of the weights

Quality loss from quantization is separate from two other costs the reader may confuse it with. KV cache growth with context length is a memory topic, not a quality one, and is covered by the context length cost page. Runtime overhead and offload slowdowns are performance topics, explained on the CPU offload performance page. None of those change the weights; only quantization does.

What this page does not tell you

This page does not give a quality score, percentage, or benchmark result for any tier. Such numbers would be fabricated here; measure them with the self-check above.
It does not recommend a tier as best for a given model. The quantization comparison page covers the tradeoff; this page grades the claims about it.
It does not cover speed, tokens per second, or hardware cost. Those depend on the engine build, driver, and offload ratio, and are out of scope.
It does not list specific cards to buy. The GPU VRAM reference and the local LLM hardware guide handle memory budgets without purchase advice.
It does not address fine-tuning quality loss, which follows different rules and is covered by the fine-tuning VRAM page.

More reading: the run locally hub links the full set of pages, and the 70B local rundown shows which tiers are realistic on common hardware. For checking actual memory once a tier is chosen, the measure VRAM usage page explains how to read what the engine reports.

Asked before trying it

Does lower quantization always mean lower quality?

The loss is directional, not a single number. Lower bits tend to hurt instruction following and long reasoning before they move perplexity, and the effect is stronger on small models. Treat the tradeoff as task-dependent, not as a fixed percentage.

Can a benchmark tell me exactly how much quality a tier loses?

No single benchmark can. Different tasks react at different rates, eval sets carry their own bias, and sampling settings shift results. A public benchmark supports broad directions, not a precise per-tier penalty.

Is Q4_K_M safe for everyday chat?

Q4_K_M is the common default because it keeps most behavior intact on most models while roughly quartering the 16-bit size. The residual behavioral change is usually small, but it is not zero and varies by model and task.

Why are small models more affected by low bits?

A smaller parameter count leaves less redundancy for quantization to spread its error across, so the same bit reduction removes relatively more representational capacity. This is reported broadly by the community and is consistent with how quantization math behaves.

How do I check quality loss on my own machine?

Pick a fixed set of prompts, run the same model at two tiers, and record the differences you can reproduce across runs. Keep temperature and seed fixed where the engine allows, and judge on the tasks you actually care about rather than on a leaderboard.

Are K-quant _M tiers better than older formats at the same bit width?

The _M variants mix block sizes so medium layers keep slightly more precision, and that tends to be gentler at the same average bit count. This is stated by the format authors and reported widely, though it is a qualitative improvement rather than a scored gain.