KV Cache Size Explained: The Formula Behind Context Length

Why long context eats VRAM: the KV cache formula, how GQA shrinks it, and a worked table showing KV cache doubling when you double the context length.

By VRAM Fit editorial Published 2026-09-16 Updated 2026-09-17

When a model runs out of memory at a long context but handled a short one fine, the KV cache is usually the reason. This page shows the formula, why context length is the dominant lever, and how modern attention designs quietly shrink the cache.

The core formula

The KV cache size in bytes is:

KV cache (bytes) = 2 × layers × kv_heads × head_dim × sequence_length × bytes_per_element

The leading 2 covers the separate K (key) and V (value) tensors. The product kv_heads × head_dim is the size of one key or value vector. sequence_length is the number of tokens held in context, and bytes_per_element is the precision the cache is stored at.

You can also write it per token, which drops the sequence length:

KV cache per token (bytes) = 2 × layers × kv_heads × head_dim × bytes_per_element

Multiply that by the context length to get the full cache. This per-token form makes the linear relationship obvious: every extra token adds the same fixed amount.

The “doubling” property

Because sequence length is a straight multiplier, halving or doubling the context halves or doubles the KV cache. The weights do not change when you change context; only the cache does. That is why a model that fits at 4k context can fall over at 32k even though the file on disk is identical.

The table below works the per-token number for three real attention shapes, then shows the full cache at four context lengths. All use 16-bit (2 bytes per element) for the cache.

Model shape	Layers	KV heads	Head dim	Per-token KV (bytes)	4k ctx	8k ctx	16k ctx	32k ctx
Small, MQA (1 KV head)	32	1	128	16,384	0.07 GB	0.13 GB	0.27 GB	0.53 GB
Mid, GQA (8 KV heads)	40	8	128	1,310,720	5.37 GB	10.7 GB	21.5 GB	42.9 GB
Large, GQA (8 KV heads)	80	8	128	2,621,440	10.7 GB	21.5 GB	42.9 GB	85.9 GB

Read the rows left to right: each step doubles the context and doubles the cache. The small MQA model stays tiny because it has a single KV head; the GQA rows show how a modest head count still produces multi-gigabyte caches at long context.

Where GQA actually helps

Grouped Query Attention was adopted widely because it attacks exactly the kv_heads term. A model with 32 attention heads but only 8 KV heads stores one quarter as many key and value vectors per token as a full multi-head design. The quality stays close because the query heads still read from the shared KV heads.

Multi-Query Attention takes this further with a single KV head, which is why the MQA row above is so small. The trade is sometimes a small hit to quality on certain tasks, but the memory saving at long context is large.

You can see the exact KV head count in a model’s config (often num_key_value_heads in GGUF or safetensors metadata). Plug it into the KV cache calculator for a cache-only number, or the VRAM calculator to get a per-model total rather than a rule of thumb.

KV cache versus weight memory

It is easy to assume the weights dominate. For short context they usually do. At long context the cache can catch up or pass them, especially on small models with many layers. Consider a 7B model at 4-bit: weights near 4 GB. At 4k context the cache might be a few hundred megabytes. At 128k context the cache can reach several gigabytes and rival the weights.

This is the practical warning behind the context length cost page: the budget you set for context is not a rounding error, it is a first-class memory line.

Precision of the cache itself

The formula above assumed 16-bit elements for the cache. Some engines store the KV cache at the model weight precision, others at 16-bit regardless, and a few support a dedicated KV quant. Because the cache can be large, the choice matters: dropping the cache from 16-bit to 8-bit roughly halves that term. Check your engine’s settings; the quantization comparison covers what each file format actually stores.

Why attention cost is a different curve

Do not conflate the two bottlenecks. The KV cache grows linearly with context. The attention computation, the matrix multiplies over the full sequence each step, grows roughly with the square of the context. Long context therefore costs you in two ways at once: more stored cache and more compute per token. The memory part is what decides whether the model loads; the compute part is what decides how slow it feels.

A quick check you can do by hand

Pick any model. Find layers, KV heads, and head dim from its config. Multiply 2 × layers × kv_heads × head_dim, then by 2 bytes if the cache is 16-bit. That is the per-token cache. Multiply by your planned context and divide by a billion for gigabytes. If the number surprises you, that surprise is the whole point of this page.

Cache compression: the methods that shrink the cache

Because the cache is the term that explodes at long context, a whole family of techniques exists to make it smaller. They attack different parts of the formula and are worth comparing in one place. The magnitudes below are orders of magnitude, not exact figures, because the real saving depends on the model and the engine.

Method	Saves (order of magnitude)	Cost	Who can use it
GQA / MQA (fewer KV heads)	Proportional to the KV-head ratio — 8 KV heads instead of 32 cuts the cache to one quarter; a single MQA head cuts it far more.	Baked into the model architecture; a small quality trade on some tasks. Cannot be changed at runtime.	Anyone running a GQA or MQA model — effectively free because the model already ships that way.
KV cache quantization	Roughly halves the cache for each halving of cache bits (16-bit to 8-bit is about a 2x cut).	Possible quality loss on long-context attention; only works if the engine supports cache quant and it is confirmed active.	Engines that expose a cache-precision setting.
Sliding window / sparse attention	Below the dense upper bound; the real cache tracks only the window or the attended subset, not the full context.	Attention cannot see beyond the window, which can hurt tasks needing long-range recall.	Models and engines built with windowed or sparse attention.
Paged / dynamic cache management	Does not cut the per-token cache but avoids reserving the full max context up front; memory is used only for live tokens across requests.	Needs an engine that implements paged attention; the saving is about waste, not a smaller per-token figure.	Serving engines such as vLLM or the llama.cpp server.

The first two change the formula’s numbers directly; the last two change how the memory is allocated rather than how big one token’s cache is. The KV cache calculator reports the dense upper bound, so it is the figure that sliding-window and paged engines then beat in practice.

Why cache compression and weight quantization are separate things

It is tempting to assume that quantizing a model also shrinks its KV cache. The two are independent inputs to the total and do not move together.

Weight quantization (Q4_K_M, Q8_0, and so on) reduces the bytes per parameter of the model file, which shrinks the weight term. The KV cache precision is a separate field: most engines keep the cache at 16-bit regardless of how aggressively the weights are quantized, unless a dedicated cache-quant option is switched on. A 4-bit model can therefore still carry a full 16-bit cache, so compressing the weights does nothing to the cache, and compressing the cache does nothing to the weights.

The gap shows up most sharply at long context. A 70B at Q4_K_M holds about 39 GB of weights, but at 128K context its cache is about 40 GB on a standard GQA shape — the cache has passed the weights entirely, and no weight-quant choice changes that. The site’s calculators keep weights and cache as two separate fields precisely because a single “model quant” number would hide this behavior. To shrink the cache you act on the cache levers above; to shrink the weights you act on the quant tier, and the total is the sum of both.

For how the quant tiers themselves trade quality against size, see quantization formats compared. For the head-count lever specifically, the GQA discussion earlier on this page works the same numbers from the formula side.

Asked before trying it

What is the KV cache in plain terms?

During generation the model reuses the attention keys and values it already computed for earlier tokens instead of recomputing them. Those stored tensors are the KV cache, and they occupy memory that grows with how many tokens are in the context.

Why does doubling the context length double the KV cache?

Context length is a linear multiplier in the KV cache formula. The cache stores one key and one value vector per token, so twice the tokens means twice the stored vectors and twice the bytes.

How does GQA reduce the KV cache?

Grouped Query Attention uses fewer KV heads than attention heads. Because the KV head count sits directly in the formula, cutting it from 32 to 8 shrinks the cache to one quarter without changing the layer count.

Is the KV cache ever bigger than the weights?

On small models with very long context it can be. A 7B model at 4-bit is about 4 GB of weights, but at 128k context the KV cache can reach several gigabytes and approach or pass the weight size.

Does the attention compute cost grow linearly too?

No. The cache grows linearly with context, but the attention math grows roughly with the square of the context length. Those are two different bottlenecks and should not be confused.