The -ngl flag (short for n-gpu-layers) is the main lever for fitting a GGUF model on your hardware in llama.cpp and the tools built on it. This page explains what it does, how the speed and memory change as you sweep it from zero to all layers, and shows commands you can copy.
What offloading means
A transformer model is a stack of layers. -ngl N keeps the first N layers on the GPU and runs the remaining layers on the CPU. The GPU layers use VRAM; the CPU layers use system RAM. The weights for the whole model are still read from disk into RAM, but only the GPU layers are copied into VRAM for compute.
This is the same idea as Ollama’s num_gpu. If you have tuned one, you understand the other; the Ollama OOM guide covers the equivalent knobs there.
How memory and speed move with -ngl
The table is qualitative because exact numbers depend on the model and drivers, but the shape is consistent. “VRAM” is the GPU memory used; “speed” is relative token throughput.
| -ngl setting | VRAM used | Relative speed | When to use it |
|---|---|---|---|
| 0 (CPU only) | Near zero | Slowest | VRAM far too small; offline batch only |
| A few layers | Low | Very slow | Just needs to run, speed irrelevant |
| Half the layers | Medium | Slow | Borderline fit, no other option |
| Most layers | High | Fast | Model nearly fits the card |
| All layers | Maximum | Fastest | Model and KV cache fit comfortably |
The key insight is the cliff between “most” and “all”. A model with most layers offloaded still shuffles data to the CPU every token, so it is much slower than the same model with every layer on the GPU. The goal is therefore to offload as many layers as fit, not merely “enough to load”.
Finding the layer count
You need the model’s total layer count to know what “all” means. It is in the GGUF metadata (often n_layers) and printed by llama.cpp at load time. A 7B model commonly has 32 layers; a 70B model commonly has 80. Enter those into the VRAM calculator to estimate the VRAM per layer.
Command examples
Load a 7B model with all layers on the GPU, 4096 context:
./llama-cli -m model.Q4_K_M.gguf -ngl 32 -c 4096Let the engine decide how many layers fit, by requesting more than exist:
./llama-cli -m model.Q4_K_M.gguf -ngl 999 -c 4096Run mostly on CPU with a little GPU help on a small card:
./llama-cli -m model.Q4_K_M.gguf -ngl 10 -c 2048For a 70B model on two GPUs, set the split and offload all layers:
./llama-cli -m model.Q4_K_M.gguf -ngl 80 -sm row -c 4096The -sm row flag spreads the model across GPUs row-wise; exact behavior depends on your build, so confirm with a memory read.
Tuning procedure
- Start at
-ngl 999and let llama.cpp take what fits. Note the layer count it actually offloads. - If it fails, lower
-nglin steps of 4 until the load succeeds. - Watch VRAM with your system monitor to see the headroom.
- Once stable, raise
-c(context) only if memory allows; context eats KV cache as explained on the context length page.
Common mistakes
- Setting
-nglto exactly the layer count but forgetting the KV cache, which pushes the card over its limit. Leave a gigabyte or two free. - Assuming partial offload is “almost as fast” as full. It is not, because of the per-token memory transfers.
- Forgetting to re-measure after a driver update. A layer count that fit last week can overflow once an engine upgrade changes its buffer size.
- Raising context after a successful load and hitting OOM on the next request, because the KV cache grew.
Offload is a budget you spend between speed and fit. Spend it on layers first, then on context, and keep a small reserve for the engine itself.
How to decide the layer split for your card
The right -ngl value is not a fixed number; it depends on the gap between your model’s need and your card’s usable memory. The decision starts with a comparison, not a guess.
First estimate the model’s footprint: the weight block from the quant tier (see the VRAM calculator) plus the KV cache for the context you intend to run (see the KV cache calculator). Then compare that total against the memory your card actually makes available to the GPU. Note that nominal VRAM is not the same as usable VRAM — the driver and display output hold back part of it, so the working budget is below the sticker number. The GPU VRAM reference lists nominal sizes; treat them as ceilings, not the amount you can fill.
| Your card’s usable memory vs the model’s weight + KV need | Suggested split | Cost direction |
|---|---|---|
| Covers all layers, the full KV cache, and headroom | Offload every layer (-ngl = full count) | Fastest; no CPU compute, no per-token transfer |
| Covers most layers but the KV cache pushes past the limit | Offload most layers, leave room for context | Fast with a small dip; context is the thing you give up |
| Weights alone nearly fill the card | Partial offload | Slow; pays the transfer tax described below |
| Weights exceed the card entirely | All CPU (-ngl 0) | Slowest; fits almost anything but only for batch or offline use |
The guiding principle is to push as many layers onto the GPU as fit with the KV cache still resident, then stop. Settling for “enough to load” while leaving layers on the CPU is the mistake that makes a borderline model feel broken rather than merely full.
Why partial offload is slower than the fraction suggests
A model split between GPU and CPU is cut at a layer boundary. On every generated token, the data that crosses that boundary has to move between VRAM and system RAM: the offloaded layers’ weights are fetched from system memory for compute, and the resulting activations are passed back so the next GPU layer can continue. That round trip happens once per token, and it happens for the whole sequence.
The penalty is therefore paid on every token, repeatedly, which is why moving 30 percent of the layers off the GPU does not cost 30 percent of the speed. The dominant cost is the memory movement itself, and system memory bandwidth sits roughly an order of magnitude below VRAM bandwidth. The CPU offload performance page works through that bandwidth relationship; the point here is that the transfer is per-token, so the tax compounds over the sequence rather than being a one-time copy.
This is also why the cliff between “most layers on GPU” and “half on CPU” is so sharp. A few boundary layers may cost little, but once a meaningful fraction lives on the CPU, every token crosses the slow path. The practical reading is unchanged: keep as many layers on the GPU as fit, not merely enough to load.
The KV cache still needs a home
-ngl moves the weights, but it does not move the KV cache, and the cache is a separate claim on the card. In most engines the KV cache is held on the GPU alongside the on-GPU layers, so the room you free by offloading weights to CPU is exactly the room the KV cache competes for. That makes -ngl and -c two ends of one budget: more layers on the GPU leaves less space for a long context’s cache, and a longer context can force layers back to CPU.
The location of the KV cache matters for behavior as well as size. If the cache were forced off the GPU it would pay the same bandwidth tax as offloaded weights on every token, so engines keep it on the GPU by default. The practical consequence for a small card is that the on-card budget is the sum of on-GPU layers plus the on-GPU KV cache, not either one alone. The KV cache size explained page covers how that cache is sized, and the context length cost page shows how fast it grows as the window lengthens.
How to confirm the offload actually took effect
After changing -ngl, the goal is to read whether the engine did what was asked, not to measure speed. Watch the following, in order:
- The startup log lines that report how many layers were offloaded. llama.cpp prints the layer split at load time, including how many layers went to the GPU and how the model was divided across devices. Confirm that number matches what
-nglrequested (or what-ngl 999decided on its own). - The VRAM figure printed when the model loads. It should sit below your card’s usable limit with room left for the KV cache, not pinned at the ceiling.
- The reported context size and the KV cache size printed alongside it, so the context you asked for is the context that was actually allocated.
- A live GPU memory reading from the system monitor after the load settles, to confirm the engine’s printed number matches reality on the hardware.
- If the model spans more than one card, the per-device allocation lines, to see how the weight block was divided.
If the offloaded layer count is lower than expected, a leftover allocation from a previous run or another process is likely holding part of the card. A clean restart followed by the same reading is the way to tell a real limit from a stale one. The measure VRAM usage page lists how to read actual memory on each platform so the printed and observed numbers can be checked against each other.