Multi-GPU VRAM: How the Model Actually Gets Split

How a model's memory is divided across several GPUs: tensor parallel, pipeline parallel and layer offload, what each costs, and why two cards' VRAM does not simply add up.

By VRAM Fit editorial Published 2026-09-17 Updated 2026-09-17

Most single-card out-of-memory errors are about weights. Most multi-card surprises are about the split itself: the model fits only because memory was divided, and the way it is divided decides both whether it runs and how slowly. This page explains how a model’s VRAM is actually partitioned across several GPUs, what each partitioning method costs, and why two cards’ memory does not simply add up to twice the capacity.

Three ways the memory gets divided

A model that does not fit on one card can be spread in three distinct ways. They are often combined, but each behaves differently.

Method	How VRAM splits	What it costs
Tensor parallel (TP)	Weights of every layer are sharded across the GPUs; each card holds 1/N of every layer’s parameters and its share of the per-layer KV cache.	Communication-intensive: every token passes activations between cards at each layer, so it needs fast interconnect such as NVLink or a wide PCIe link to stay efficient.
Pipeline parallel (PP)	Layers are divided into contiguous stages; each card owns a block of layers and their activations.	Idle bubbles: while one stage computes, the others wait, so throughput depends on enough micro-batches to keep the pipeline full. Memory is cheap to split; utilization is the price.
Layer offload to a 2nd card or host memory	A subset of layers lives on a slower device or system RAM; only the active layers sit on fast VRAM.	Largest speed loss: every offloaded layer is fetched per token, dominated by memory bandwidth rather than compute, so latency grows with each offloaded layer.

Tensor parallel keeps every card busy on the same token but moves the most data between them. Pipeline parallel moves the least data but leaves cards idle between stages. Offload moves the model out of VRAM entirely for the layers it touches, which is why it costs the most speed. The choice is a trade between communication, idle time, and memory bandwidth.

The KV cache must be split too

The cache is the item most often left out of a multi-GPU plan. It scales with context length, just as on a single card, and it has to live somewhere on the divided model.

Under tensor parallel, each card holds its slice of the per-layer cache, so a longer context raises every card’s memory in step. Under pipeline parallel, the cache for a layer lives on the stage that owns that layer, so context length inflates the memory of whichever stages are active. Either way, the cache is not free memory on top of the split — it is part of the split. A deployment sized only for weights will run out of memory at long context even though the weights fit, because the cache grew on cards that were budgeted for weights alone.

This is why the per-token cache figure matters as much on multi-GPU as on single-GPU. A 7B at Q4 weighs about 3.9 GB whether on one card or eight, but its 128K cache adds roughly 16 GB that the split must also carry. The site’s KV cache calculator reports the per-token figure; the same number applies to each slice of the split.

Two cards’ VRAM does not simply add up

A second GPU is not a second copy of the first card’s capacity. Three effects shrink the practical total below the nominal sum.

Each card keeps its own overhead. The driver, the engine context, and the activation baseline are reserved per device. Two 24 GB cards do not yield 48 GB of model memory; a slice of each is already spoken for before the model loads.
Mixed cards run at the slowest pace. When two different models are combined, the card holding the slower or smaller slice sets the limit for what it owns. The larger card cannot compensate for a smaller card’s memory cap on the layers assigned to it, and a slower card drags the tokens it processes down to its speed.
Communication costs memory too. Splitting adds activation buffers, all-reduce scratch space, and queue memory that did not exist on a single card. The interconnect bandwidth decides how much of that overhead turns into latency versus just memory use.

The usable total is therefore below the sum of the labels. The GPU reference lists nominal capacities; the Apple Silicon unified memory page covers the case where one pool is shared rather than split. The run 70B locally guide shows what a 70B actually needs once these effects are counted.

Decide the card count from the demand

The reliable order is to compute first, then divide.

Total demand — weights (from the quant) plus the KV cache for the intended context plus a per-card and per-engine overhead, and multiplied by the expected concurrent sequences if serving users. The context length cost page shows how context and batching grow that demand.
Per-card usable memory — nominal capacity minus the overhead each card reserves, using the slower or smaller card as the bound when mixing.
Minimum card count — total demand divided by per-card usable memory, rounded up, then add one for headroom because peaks exceed the steady state.
Pick the split method — tensor parallel when interconnect is fast and the model is wide, pipeline parallel when stages map cleanly and bubbles are acceptable, offload only when no faster split fits.

Starting from the card count and hoping the model fits inverts the logic and produces the exact surprise this page describes. The local LLM hardware guide frames the same decision from the buying side: size the memory first, choose the cards after.

Offload as the capacity backstop

Layer offload to a second card or to host memory is the method that recovers the most capacity at the highest speed cost. A layer kept in system RAM is fetched every time it is used in the forward pass, so the cost is paid per token, not per load. This is the trade the CPU offload performance page quantifies: capacity won back, latency paid back continuously.

Offload is the right call when the model is too large for any affordable split but the workload tolerates slower replies. It is the wrong call when throughput matters, because the per-token fetch dominates. Treat it as the backstop after tensor and pipeline parallel have been tried, not the first option.

What this page does not tell you

It does not give framework-specific configuration. The commands and flags for a given engine change between versions, so this page describes the memory behavior rather than the exact settings to type.
It does not guarantee that any combination of cards runs a given model. Real behavior depends on the engine, the interconnect, the quant, and the context; the splits here are the memory shape, not a promise.
It does not provide measured interconnect bandwidth numbers. NVLink, PCIe, and host-memory bandwidth vary by generation and topology, and the speed cost of a split follows from those measured figures, which are outside this page’s scope.
It does not cover distributed training state such as optimizer and gradient memory, which follow the fine-tuning VRAM multipliers rather than the inference split described here.

For the single-card baseline these splits build on, see the VRAM calculator. For how concurrency multiplies the demand the split must meet, see batch size and throughput. The llama.cpp offload layers page covers the offload path in one engine, and the run locally hub collects every page in the site.

Asked before trying it

Does adding a second GPU double the usable VRAM?

Not literally. Each card carries its own driver and engine overhead, the combined pool is limited by the slowest card when models differ, and the splitting itself adds communication buffers. The practical usable total is below the nominal sum of the two cards.

What is the difference between tensor parallel and pipeline parallel?

Tensor parallel shards each layer across GPUs so every card holds a slice of every layer; it is communication-intensive. Pipeline parallel splits the model into stage blocks of layers, one block per card; it trades memory for idle bubbles while stages wait on each other.

Does the KV cache also get split across GPUs?

Yes, and it is the item most often missed. The cache scales with context length, so under tensor parallel each card holds its share of the per-layer cache, and under pipeline parallel the cache lives on the stage that owns those layers. Long context raises the memory the split must cover.

When should layers be offloaded instead of buying a bigger card?

Offloading trades speed for capacity. A layer kept on host memory or a slower card must be fetched every token, so it costs the most latency of the three methods. It is the option of last resort when no single card and no faster split fits the model.

How many GPUs are needed for a given model?

Compute the total demand first: weights plus KV cache plus overhead for the intended context and batch. Then divide that total by the per-card usable memory, after overhead, to get the minimum card count. Decide the number of GPUs from the demand, not the other way around.

Can different GPU models be combined?

It is possible to combine cards, but the slower card sets the pace for the layers it holds, and mismatched memory caps the split at the smaller card's limit for the slices it owns. Mixed setups work but rarely reach the sum of their specs.