A model that loads cleanly for one person can still run out of memory the moment several people use it. The reason is not the weights. The weights are loaded once and shared. The part that grows with usage is the KV cache, and each concurrent session keeps its own copy. Serving a model is therefore a different memory problem from chatting with it, and planning for one does not prepare for the other. This page estimates how the KV cache scales with concurrency and explains when a deployment has to reserve VRAM specifically for that growth.
KV cache is one copy per concurrent sequence
The KV cache stores a key and a value vector for every token in a sequence, so the model does not recompute attention from scratch as the context grows. That storage is per sequence. When one person chats, there is one cache. When ten people chat at the same time, there are ten caches, one for each live session.
The weights sit outside that multiplication. A 7B or 8B model’s weights are loaded a single time and read by every session; the GPU does not duplicate them per user. This is the central point: concurrency multiplies the cache, not the model. A deployment with ten active sessions pays ten times the cache and one times the weights, which is a very different budget from ten times the whole model.
The cache per session is itself set by two things: the model’s attention shape and the session’s context length. Two sessions at 4K context each hold a small cache; two sessions at 128K each hold a cache that can be tens of gigabytes apiece. The context length chosen for the service sets the per-session cost, and concurrency then multiplies that cost.
The concurrency growth table (estimate)
The table below estimates total KV cache for a representative 8B-class model at 128 KB per token, using the per-context figures from the site’s shared model reference: about 0.5 GB at 4K, 4 GB at 32K, and 16 GB at 128K per single session. Multiply by the number of concurrent sessions. These are formula estimates, not measured readings; the weights are added once on top.
| Concurrent sessions | 4K context | 32K context | 128K context |
|---|---|---|---|
| 1 | 0.5 GB | 4.0 GB | 16 GB |
| 2 | 1.0 GB | 8.0 GB | 32 GB |
| 4 | 2.0 GB | 16 GB | 64 GB |
| 8 | 4.0 GB | 32 GB | 128 GB |
| 16 | 8.0 GB | 64 GB | 256 GB |
| 32 | 16 GB | 128 GB | 512 GB |
For other models, scale the per-session figure by the ratio of their per-token KV cost: a 3B model is about 0.875× of the 8B figure, a 14B model about 1.5×, a 32B model about 2×, and a 70B model about 2.5×. The shared weights are not scaled by concurrency, only added once. A real service total is therefore “weights + the row value above for the chosen context and peak session count.” The KV cache calculator lets you set a model and length directly, and the context length cost page shows how a single session’s cache climbs with length before concurrency is even considered.
Why weight-fits does not mean service-runs
It is common to confirm a model loads in Ollama or llama.cpp with one session, see a comfortable ollama ps reading, and assume a small server will work. The single load only proves the weights plus one cache fit. Under real use, sessions open and close continuously, and at peak several are live together. Each adds its own cache, and the combined cache can exceed the free VRAM even though no single session would have.
The failure mode is distinctive: the model starts fine, answers the first few requests, and then throws an out-of-memory error once enough sessions stack up. That is a concurrency symptom, not a weight symptom. The fix is not to reload the model or pick a smaller one outright, but to cap the context, cap the concurrent sessions, or reserve enough VRAM for the peak cache. The engine comparison page lists how llama.cpp, Ollama, and vLLM differ in how they handle this peak, because their schedulers make different trade-offs about when to admit new sessions.
The wrong intuition: concurrency 10 is not model times 10
A frequent misread is that ten concurrent users means ten times the VRAM. That is true only for the KV cache. The weights are shared, so the model term appears once, not ten times. For an 8B model at Q4_K_M, the single-session budget at 4K is roughly 5.5 GB of weights plus 0.5 GB of cache, about 6 GB. Ten concurrent 4K sessions are about 5.5 GB of weights plus 5 GB of cache, about 10.5 GB — under two times the single load, not ten. The bigger the model and the longer the context, the more the shared weights dominate and the less concurrency multiplies the total in relative terms, even though the absolute cache growth is still linear and still real.
The practical consequence is that small models gain the most, proportionally, from concurrency headroom, because their weights are a smaller share of the total. A 70B model’s weights already dominate, so adding sessions mostly adds cache on top of a heavy base. Either way, the planning must use weights-once-plus-cache-times-sessions, never model-times-sessions.
When you must reserve VRAM for concurrency
The table below is a judgment aid, not a hard threshold. It maps a deployment shape to the memory it should reserve. The key is to size for peak simultaneous sessions, not the average, because the cache allocates per live sequence and the worst case is the busy moment, not the quiet one.
| Deployment | Typical peak concurrency | What to reserve | Why it matters |
|---|---|---|---|
| Single-user chat | 1 | weights + one session’s KV | Only one cache ever exists; the weights dominate the budget. |
| Small team or few users | 2 to 8 | weights + KV × peak sessions | Each active session holds its own cache; plan for the busiest moment. |
| External or public service | 10 and up | weights + KV × peak + scheduler headroom | Peak, not average, sets the limit; the scheduler needs room to admit and queue. |
A single-user setup can ignore concurrency almost entirely and spend its whole budget on weights and a comfortable context. A small team should reserve for the number of people likely to be active at once; a few concurrent 8K or 16K sessions can add several gigabytes that a one-at-a-time test never shows. A public service has to treat peak concurrency as a first-class capacity input and usually cap either the context or the session count to stay within the card. The local LLM hardware guide frames how to choose a card by the same budget reasoning, and the CPU offload performance page shows what happens to speed when layers leave the GPU to make room.
Trading memory for throughput, in order
When a serving setup is short on VRAM, the recovery should follow a deliberate order that protects output quality for as long as possible.
- Lower the context window first. Context is the cheapest thing to lose for most workloads. Halving the context roughly halves the per-session cache, and it usually costs little because real exchanges are short. This is the same move that rescues an out-of-memory load.
- Lower the KV cache precision next. Some engines can store the KV cache at a reduced precision instead of the default 16-bit, shrinking the cache substantially per session. This trades a small amount of attention fidelity for a large memory win and leaves the weights intact.
- Lower the weight quant tier last. Dropping from Q4_K_M to Q3_K_M or Q2_K reduces the weights directly, but it is the step most likely to change output behavior, so it comes after the cache levers are exhausted.
That order is the reverse of what a naive fix does, which is to reach for a smaller model immediately. A smaller model changes everything; trimming context and cache precision often recovers enough room to keep the same model. The quantization formats compared page lists the weight tiers, and the quantization quality loss page describes what each step costs in output terms, with source grades kept honest.
Scheduling and cache management are engine details
How an engine actually holds and reuses these caches is an implementation matter. A naive server allocates a full cache per session up front; more advanced servers carve cache dynamically, reuse freed blocks, and share prefix context across sessions that share a prompt. Those techniques change the real memory footprint in ways the simple per-session multiplication here does not capture. This page deliberately stops at the linear per-session estimate, because that is the floor any deployment must meet regardless of scheduler cleverness. The engine comparison page contrasts how the major runtimes behave, and the measure VRAM usage page explains how to read the true allocated total on each platform so the estimate can be checked against reality.
What this page does not tell you
- It does not give measured throughput numbers. Generation speed depends on the build, the driver, the batch shape, and the offload ratio, none of which this page estimates.
- It does not cover differences between scheduler implementations. How vLLM, Ollama, and llama.cpp admit and manage sessions changes the real footprint.
- It does not quantify the specific savings from dynamic cache management such as paged or block-based allocation. Those techniques reduce the gap below the linear estimate but are not measured here.