Ollama reports out of memory when a load or a generation asks for more VRAM than is free. The fix is almost never a reinstall; it is matching the model, context, and offload settings to the card you have. This page gives a symptom table and explains the two settings that matter most.
The two settings that control memory
Ollama exposes its memory behavior through model options. The two with the largest effect are:
num_gpu— the number of layers kept on the GPU. Each layer left on the GPU uses VRAM but runs fast; each layer moved to CPU uses system RAM instead and runs slow. Raisingnum_gputo the maximum your card allows is fastest; lowering it trades speed for fit.num_ctx— the context window in tokens. The KV cache scales linearly with this number, so a highnum_ctxcan quietly consume several gigabytes. This is the setting people forget.
A typical invocation looks like:
ollama run llama3:70b-instruct-q4_K_M -o num_gpu=48 -o num_ctx=4096If it fails, drop num_ctx before you drop num_gpu, because context is usually the cheaper thing to lose.
Symptom to root cause table
| Symptom | Likely root cause | What to do |
|---|---|---|
| Model refuses to load at all | Weights exceed free VRAM | Lower num_gpu, or use a lower quant tier from the quantization comparison |
| Loads, then OOM on first long reply | KV cache for large num_ctx too big | Reduce num_ctx to shrink the cache |
| Worked yesterday, fails today | Stale VRAM from a prior run or another process | Restart Ollama; close other GPU apps |
| OOM only with multiple concurrent chats | Batching multiplies KV cache | Lower num_ctx or limit concurrency |
| Fails on a 24 GB card at Q4 | Model simply too large for one card | Offload more layers, use two GPUs, or go smaller model |
| Slow but no OOM after lowering num_gpu | Too many layers on CPU | Raise num_gpu until it just fits |
Steps that reliably free memory
- Restart the Ollama service. This releases VRAM held by a previous load that did not clean up.
- Lower
num_ctx. Cutting context from 32768 to 4096 can free several gigabytes on its own. - Lower
num_gpugradually. Each step moves layers to system RAM, trading speed for fit. - Pick a lower quant tier. Moving a 70B from Q4_K_M to Q3_K_M drops the weights by roughly a quarter.
- Close other GPU users. A browser with hardware acceleration or a second model server can hold gigabytes you need.
- Verify with a memory read using your system’s monitoring tools to see what is actually allocated.
Why the model sometimes loads and sometimes not
A model near the limit is unstable by definition. Between runs, the driver may not have released every allocation, or a background process grabbed part of the card. The same file that loaded cleanly at 10 AM can throw OOM at 10:05 with a slightly larger context. Treat any model that sits within a gigabyte of your card as unreliable under load, and either offload more or downsize. A clean restart between attempts is the single most reliable fix for a borderline model.
Offload is the same idea as llama.cpp
Ollama’s num_gpu and llama.cpp’s -ngl are the same mechanism: keep the first N layers on the GPU and compute the rest on CPU. The offload layers guide explains how the split changes speed and memory in detail. If you tune one, you understand the other.
A quick triage order
When you hit OOM, do not guess. Restart, then re-run with num_ctx=2048. If it loads, raise context until it stops. If it still fails, lower num_gpu. If it still fails, the weights themselves are too big and the only real fixes are a smaller model, a lower quant, or more total memory as described on the 70B local page.
An ordered checklist to recover from OOM
The steps below are an execution order, not a symptom list. The order matters because each earlier step costs less in speed or quality than the ones after it. Start at the top and stop as soon as the model loads and stays loaded.
| Step | Action | Cost direction | Why this position |
|---|---|---|---|
| 1 | Restart the Ollama service | Free — releases stale VRAM | Clears leftover allocations before anything else changes |
| 2 | Lower num_ctx | Low — smaller context window | The KV cache shrinks roughly in proportion; often enough on its own |
| 3 | Close other GPU apps | Free — recovers held memory | Browsers and second servers can hold gigabytes you need |
| 4 | Lower num_gpu | Trades speed for fit | Moves layers into system RAM, the first real offload step |
| 5 | Pick a lower quant tier | Trades quality for fit | Smaller weights, but changes output behavior |
| 6 | Smaller model or more memory | Largest change | Only when every lever above is exhausted |
Step 2 before step 4 is the key ordering: context is usually the cheaper thing to lose, because it scales the KV cache without touching the weights. Step 5 is near the end on purpose — dropping quantization is the move most likely to change how the model answers, so it comes after the memory-only levers. This list is the same recovery the triage order uses, laid out with the cost of each move so the trade is visible before it is made.
Where the memory goes at load time
An out-of-memory error at load is easier to reason about once the three sinks are separate. At the moment a model loads, memory is claimed by three things, and they do not move together:
- Weights. The resident parameter block, set by the parameter count and the quantization tier. This is the largest line and the one people size against.
- KV cache. Allocated for the context window you asked for. It scales with context length and, in serving, with concurrent sessions, and it is typically held at 16-bit regardless of the weight quant.
- Engine reserve. A smaller hold for the runner, the driver, and the kernel buffers that sit beside the model. This is the part most often forgotten.
The surprise is that “the weights fit” is not the same as “the model fits.” A model whose weight block reaches 90 percent of the card leaves no room for the KV cache of a real context or for the engine reserve, so it loads and then throws OOM on the first long reply when the cache has to grow. The Ollama VRAM requirements page builds the same three terms into a formula estimate, and the VRAM calculator adds them for a specific model. The takeaway for troubleshooting is that the free memory must cover weights plus the cache you actually intend to use, not just the weights.
When to stop tuning and switch models
There is a point past which further tuning only wastes time. Consider switching models when:
- The model will not load even after a clean restart, the smallest usable
num_ctx, and the lowest practical quant tier that still meets your quality bar. - The only
num_ctxthat fits is shorter than your real workload needs. If you need a 32K window but only a 2K window loads, the model is not fit for the task even though it technically runs. - Lowering
num_gputo its floor still overflows, meaning the weight block alone exceeds the card.
At that point the realistic options are a smaller model, a lower quantization, or more total memory through a second card or a unified-memory machine — not another round of the same knobs. The 70B local page and the multi-GPU page describe how bigger models are actually hosted, and the Apple Silicon unified memory page covers the shared-pool alternative. The decision is about how much memory the model needs, not which card to buy.
How to verify after changing settings
A setting change is only proven if it is checked the same way each time. Keep the comparison clean:
- Use the same prompt and the same
num_ctxbefore and after the change, so the KV cache term is identical and the only variable is the lever you moved. - Read the actual memory with
ollama psafter the model loads, and compare the reported VRAM against the previous reading. - Repeat the load a few times rather than trusting a single success. A borderline model can load once and fail on the next attempt when a slightly larger context arrives, so reliability across attempts is the signal, not one green load.
- Judge responsiveness qualitatively for your use case rather than by any speed figure. The question is whether the model stays loaded and answers within the window you need, not how many tokens per second it produces.
The measure VRAM usage page lists how to read the true allocated total on each platform so the before and after numbers come from the same source. A change that lowers the ollama ps reading and survives several loads has actually worked; a change that loads once and then fails again has only shifted the border.