CPU Offload: How Much VRAM You Save, and What It Costs

CPU offload trades VRAM for speed. This page shows the tradeoff by offload ratio and why the slowdown is not linear, plus the order to try other levers first.

By VRAM Fit editorial Published 2026-09-17 Updated 2026-09-17

CPU offload is the lever that lets a model load when it does not fit on the GPU. It does that by moving part or all of the weights out of VRAM and into system RAM, where the CPU computes them. The benefit is clear: VRAM drops. The cost is also clear but less intuitive: generation slows, and it slows by more than the fraction of layers moved would imply. This page is about that tradeoff — how much VRAM each offload level saves, what happens to speed, and the order in which offload should be considered against the other ways to fit a model.

What offloading actually moves

A model’s weights are the bulk of the memory it needs. By the site’s standing formula, weight VRAM is the parameter count in billions times the bits per parameter divided by eight. When layers are offloaded to the CPU, those layers’ weights leave VRAM and live in system RAM instead. The GPU still does the layers kept on it; the CPU does the rest. The whole model is still read from disk into RAM, but only the on-GPU layers are copied into VRAM for compute.

This is the same idea behind the -ngl flag in llama.cpp and num_gpu in Ollama. The page on llama.cpp offload layers explains the flag mechanics and how to sweep it. This page takes the next step: it treats offload as a budget spent between VRAM and speed, and asks when the spend is justified.

The tradeoff by offload ratio

The table below is qualitative by design. Exact numbers depend on the model, the driver, the build, and the offload ratio itself, so they are not stated here. What stays constant is the shape: as more layers leave the GPU, VRAM falls and speed falls, but speed falls faster than the layers would suggest.

Offload configuration	VRAM saved (magnitude)	Speed direction	When it is still worth doing
All layers on GPU	None	Fastest	Model and KV cache fit comfortably; this is the target state
Mostly on GPU, a few layers on CPU	Small fraction of weights	Fast, slight dip	Only a hair over the card; nearly ideal
About half the layers on CPU	Around half the weight VRAM	Slow	Borderline fit with no other option; a stopgap, not a plan
Most layers on CPU	Most of the weight VRAM	Very slow	The model would not load otherwise; batch or offline use only
Entirely on CPU	Nearly all weight VRAM	Slowest	Nothing fits on the GPU at all; throughput is the lowest tier

The headline is the gap between “mostly GPU” and “half on CPU”. A model with half its layers on the CPU does not run at half the speed of the all-GPU version. It runs much worse, because of what happens on every single token.

Why the slowdown is not linear

The non-linear slowdown comes from bandwidth, not from compute alone. The weights that were moved to the CPU still have to participate in every token. For each generated token, the engine must pull the offloaded weights across the boundary between system RAM and VRAM, compute them, and pass the result back. This is a memory-movement problem before it is an arithmetic problem.

System memory bandwidth is about an order of magnitude lower than VRAM bandwidth. A discrete graphics card’s VRAM is built for high-throughput, contiguous transfers; the host memory path is far narrower and goes through the CPU interconnect. So the per-token transfer of the offloaded weights dominates the generation timeline. Losing half the layers can cost far more than half the speed, because the cost is paid on every token, repeatedly, for the whole sequence.

This is why partial offload is a sharp cliff rather than a smooth slope. The first few layers moved to the CPU may cost little, but once a meaningful fraction crosses the boundary, every token pays the transfer tax. The practical conclusion is to keep as many layers on the GPU as fit, not merely “enough to load”.

The order to pull other levers first

Offload is the most expensive lever in speed terms. Before moving weights to the CPU, the cheaper levers should be exhausted, because they change the budget with less damage to generation speed.

Reduce the context length first. The KV cache grows linearly with context and is pure overhead on top of the weights. Dropping from 32k to 8k context can free several gigabytes with no change to the weights and only a modest effect on speed for short exchanges. The context length cost page shows how the cache scales.
Look at the KV cache precision. The cache is typically held at 16-bit and is independent of the weight quantization. If the engine supports a lower cache precision, that reduces a separate, often large, line item without touching the weights.
Consider offload only after the above. Offload trades the largest speed penalty, so it comes third in the order.
Last, reduce the quantization level. Dropping from Q4_K_M to Q3 or Q2 reduces the weight size directly, but it is listed last because lower quantization changes output quality. The quantization comparison and the quality loss page cover that trade.

This order puts the smallest speed and quality costs first. Offload is reached only when the cheaper moves still leave the model unloadable.

“Can run” and “can be used” are not the same

A model that offloads to the CPU will complete. That is not the same as being practical to use. At the fully CPU-offloaded tier, generation is slow enough that interactive chat, agent loops, and tool-using flows may be unusable even though every request eventually returns. The process runs; the workflow does not move.

The distinction matters for planning. A user who needs a model to answer a handful of prompts overnight tolerates a slow, offloaded run. A user who needs a responsive assistant or an agent that calls tools repeatedly does not. The same offloaded configuration can be acceptable in one case and unusable in the other. Sizing should target the use case, not merely the load.

The unified-memory exception

On Apple Silicon the CPU and GPU share one memory pool, so there is no separate system RAM sitting across a narrow interconnect from the VRAM. Offloaded weights move within the same pool rather than across a bandwidth cliff. The per-token transfer penalty is therefore much milder than on a discrete card with separate RAM.

This does not make offload free. The shared pool still has a bandwidth ceiling and the CPU compute is slower than the GPU’s, so moving layers off the GPU still costs speed. But the dramatic, non-linear collapse seen on discrete cards is softened. The Apple Silicon unified memory page covers how the shared pool changes the budget.

Estimating the saved VRAM

The saved VRAM is, to a first estimate, the weight size of the layers moved. For a model whose weights are estimated at roughly 4 GB in a given quantization, fully offloading it keeps nearly that entire 4 GB out of VRAM. The qualifier “estimated” matters: the real figure depends on the parameter count, the quantization bits, and the small overhead GGUF files carry. Use the VRAM calculator to estimate it, and the model file size lookup for representative numbers per model and tier.

Note that offload does not reduce the KV cache. The cache is allocated per context regardless of where the weights live, so any context-length savings still have to be found through context settings, not offload.

Measuring rather than guessing

Because the slowdown depends on the build and the offload ratio, the only reliable numbers come from the machine itself. Watch VRAM with the system monitor and watch generation responsiveness in a real session. The measure VRAM usage page lists how to read actual memory on different systems. A configuration that loads is only half the answer; the other half is whether it responds fast enough to be used.

If the model still overflows the card after context and cache levers are pulled, offload is the remaining option. If it loads but is too slow to use, the realistic answer is usually a smaller model or a lower quantization that keeps more layers on the GPU, not deeper offload. The local hardware guide frames the capacity question in terms of how much VRAM is needed rather than which card to buy, and the Ollama VRAM requirements page shows what common models actually consume.

What this page does not tell you

This page does not give token-per-second numbers for any offload ratio, because those depend on the build, driver, batch size, and exact layer split and cannot be stated honestly up front. It does not rank or recommend specific graphics cards to purchase, and it does not claim that offload is the right answer for any particular model — only that it is the most speed-expensive lever and should be tried after context and cache reductions. Peak memory under concurrency, multi-GPU splits, and fine-tuning offload are covered on other pages. The bandwidth figures are described as an order-of-magnitude relationship, not measured benchmarks on a specific card.

Asked before trying it

How much VRAM does moving layers to the CPU save?

Roughly the weight size of the layers moved. A model with weights estimated around 4 GB that is fully offloaded to CPU keeps nearly all of that 4 GB out of VRAM. The exact figure depends on the parameter count and quantization, so estimate it with the VRAM calculator.

Why is the speed loss not proportional to the layers left on CPU?

Every generated token must move the offloaded weights between system RAM and VRAM, and system memory bandwidth is about an order of magnitude below VRAM bandwidth. That per-token transfer dominates the timeline, so a half-offloaded model is far slower than the linear fraction suggests.

Is offload worth it if the model barely fits on the GPU?

Only when the alternative is that the model does not load at all. Offload trades a large speed penalty for the ability to run. If the goal is interactive use, a smaller quantization often keeps more layers on the GPU and ends up both faster and usable.

What should be tried before offloading to the CPU?

Reduce the context length first, then look at the KV cache precision, then consider offload, and only last reduce the quantization level. Those levers change the budget with less damage to generation speed than moving weights off the GPU.

Does offload behave the same on Apple Silicon?

Unified memory changes the picture. The CPU and GPU share one pool, so the per-token transfer between host and VRAM is far less of a penalty than on a discrete card with separate system RAM. The tradeoff is milder, though not free.

Can a model that offloads still be used for real work?

It can run, which is not the same as being practical to use. A fully CPU-offloaded model generates slowly enough that interactive chat or agent loops may be unusable even though the process completes. The line between runs and is usable is separate.