Most out-of-memory errors at load time are about weights. Most out-of-memory errors during a session are about context. This page shows how chat length turns into KV cache memory and how running several conversations at once multiplies that cost on top of the model itself.
The linear rule
The KV cache holds one key and one value vector for every token in the context. Because context length is a straight multiplier in the cache formula, the relationship is linear: double the tokens, double the cache. The weights stay fixed; only the cache moves.
This is the single most useful fact on the page. If a model loads at 4k context and fails at 32k, the weights did not change. The cache did, fourfold.
Chat length to KV cache, worked
Using a representative shape of 40 layers, 8 KV heads, 128 head dimension, and a 16-bit cache (2 bytes per element), the per-token cache is:
2 × 40 × 8 × 128 × 2 = 163,840 bytes ≈ 0.00015 GB per token
Scale that by context length:
| Context length | KV cache (this shape) | Typical use |
|---|---|---|
| 2,048 | 0.34 GB | Short Q&A |
| 4,096 | 0.67 GB | Standard chat |
| 8,192 | 1.34 GB | Long docs |
| 16,384 | 2.68 GB | Codebase snippets |
| 32,768 | 5.37 GB | Long documents |
| 65,536 | 10.7 GB | Book-length |
| 131,072 | 21.5 GB | Maximum context |
At 128k context the cache alone is larger than many graphics cards. A model that fits easily at 4k can become impossible at 128k purely from the cache, with zero change to the weights.
Batching multiplies the cache
A single chat uses one KV cache. A server handling several users at once keeps one cache per active sequence. If four people each hold an 8k context, the engine stores four caches:
4 × 1.34 GB ≈ 5.37 GB of KV cache on top of the shared weights.
The weights are loaded once and shared; the caches are not. Concurrency is therefore the silent memory multiplier that bites self-hosted servers. Plan for peak simultaneous contexts, not average, because the cache memory is allocated per live sequence.
Sizing a real session
- Estimate your longest real exchange in tokens, not characters. A few thousand words is roughly a few thousand tokens, but code and repeated text vary.
- Compute the KV cache for that length with the formula above.
- Add it to the weight size from the quantization comparison.
- Add engine headroom and, if serving users, multiply the cache by expected concurrent sessions.
- Check the sum against your card’s usable memory.
Practical defaults
- For interactive chat, 4k–8k context covers most exchanges and keeps the cache under about 1.5 GB on the shape above.
- For document question answering, 16k–32k is common; budget several gigabytes accordingly.
- For “ingest a whole book” use, 128k is tempting but the cache alone can exceed a 24 GB card. Prefer chunking or a larger memory pool on a Mac with high unified memory.
Estimating your prompt in tokens
You rarely know the exact token count until you run it. A few rules of thumb help you size context without a tokenizer open:
- English prose runs about 0.75 token per word, so 1,000 words is roughly 750 tokens.
- Code and JSON are denser, often 1.5 to 2 tokens per word, because symbols each become their own tokens.
- Repeated boilerplate, such as a system prompt pasted into every call, counts every time and quietly eats the budget.
- A long document you paste for question answering is the largest single source; count it, do not guess it.
If your estimate plus the model’s fixed overhead approaches the context you set, lower the setting rather than risk an OOM mid-reply.
The trap of maximum context
Model cards advertise a large max context, but that is a capability, not a free setting. Enabling it reserves cache memory whether or not you use it. Set the context to what your session needs and no more; the savings are immediate and often the difference between a stable run and an OOM mid-conversation.
How a real session’s memory walks up
The tables above are static snapshots. A live session grows in steps, and the cache climbs with it. The walk below uses a Llama 3.1 8B-class shape (128 KB per token at 16-bit cache, from the site facts file) and is an estimate that depends on the actual layer and head counts of the model you run.
- Empty session. With only a short system prompt loaded, the context holds a few hundred tokens, so the KV cache is a fraction of a gigabyte — well under the weights, which are fixed at load. On this shape a 500-token prompt is about 0.06 GB of cache (estimate).
- A long conversation. As the exchange passes a few thousand tokens, the cache reaches roughly the 4K figure: about 0.5 GB. The model did not change; the context simply accumulated.
- Pasting a long document. Dropping a 32K-token document into the chat adds about 4 GB of cache on top of the running conversation. The cache can now rival or pass the weight size of a small model, which is why the same model that chatted fine all afternoon suddenly throws an OOM the moment a large file is attached.
The direction is the useful part: weights stay flat, the cache is what moves, and a single pasted document can contribute more cache than the entire earlier conversation. For the exact breaking point on a given card, the context length calculator inverts the formula and reports the longest context that fits after the weights, which is the complement to the principle explained here.
Typical session length to cache occupancy
This table estimates the cache for realistic session sizes on a Llama 3.3 70B GQA shape (320 KB per token at 16-bit cache, from the site facts file). It isolates the cache only and excludes the weights and engine overhead, so the numbers are planning figures that depend on the attention shape used.
| Session scenario | Approx tokens | KV cache (70B GQA, estimate) |
|---|---|---|
| Fresh start, system prompt only | ~500 | ~0.15 GB |
| Typical back-and-forth chat | ~4,000 | ~1.25 GB |
| Long chat carrying full history | ~16,000 | ~5 GB |
| Pasted document plus chat | ~32,000 | ~10 GB |
| Book-length ingest | ~131,072 | ~40 GB |
The spread from a fresh start to a book-length ingest is roughly 250 times, driven entirely by context, with zero change to the model file. That is the range a session planning page has to reason about, and it is why the cache deserves its own line in any budget.
Prefill and token-by-token generation behave differently
The two phases of a session allocate memory on different curves, even though both end up feeding the same KV cache.
During prefill, the engine processes the entire input at once — the pasted document, the full prompt, the accumulated history. It must materialize activations for the whole sequence simultaneously, so the transient peak allocation is larger than the steady state and is set by the input length, not by the per-step cost. A long prompt therefore creates a memory spike the moment generation begins.
During generation, the model adds one token per step. The KV cache grows by exactly one fixed per-token amount each step, and the per-step activation footprint stays small because only the newest token is being computed against the already-stored cache. The cache rises steadily, but there is no second input-length spike.
The practical consequence is that peak memory usually occurs at prefill, not during generation, and the peak is driven by activations over the full input rather than by the cache alone. Planning a session needs both numbers: the steady-state cache from the formula above, and extra headroom for the prefill peak. The same physics is what the context length calculator leaves as a headroom note rather than modeling directly.