A 70B model is the first size where “just buy a bigger card” stops being a one-card answer. This page lays out what each GGUF tier needs and which common hardware arrangements can actually run it, including the multi-GPU and CPU-offload paths most people end up taking.
The weight floor for 70B
Start from parameters × bits ÷ 8 with parameters at 70. The weight memory alone, before any KV cache or engine overhead, is:
| Tier | Bits | 70B weights |
|---|---|---|
| Q2_K | 2.6 | 22.8 GB |
| Q3_K_M | 3.5 | 30.6 GB |
| Q4_K_M | 4.5 | 39.4 GB |
| Q5_K_M | 5.5 | 48.1 GB |
| Q6_K | 6.6 | 57.8 GB |
| Q8_0 | 8.5 | 74.4 GB |
| F16 | 16 | 140 GB |
Add a couple of gigabytes for the KV cache at normal context and engine overhead, and the real number sits a bit above these figures. The VRAM calculator produces the exact total for your context length.
Feasibility matrix by hardware
The matrix judges fit at a typical 4k–8k context. “Tight” means it loads but leaves little room for long context or large batches. “CPU mix” means part of the model runs in system RAM.
| Hardware | Q2_K | Q3_K_M | Q4_K_M | Q5_K_M | Q6_K | Q8_0 |
|---|---|---|---|---|---|---|
| Single 12 GB (RTX 3060) | CPU mix | CPU mix | CPU mix | CPU mix | CPU mix | CPU mix |
| Single 16 GB (RTX 4060 Ti) | CPU mix | CPU mix | CPU mix | CPU mix | CPU mix | CPU mix |
| Single 24 GB (RTX 3090/4090) | Tight | CPU mix | CPU mix | CPU mix | CPU mix | CPU mix |
| Two 24 GB (SLI-free, layer split) | Yes | Yes | Yes | Tight | CPU mix | CPU mix |
| Single 48 GB (workstation) | Yes | Yes | Yes | Yes | Tight | CPU mix |
| Apple 64 GB unified | Yes | Yes | Yes | Tight | CPU mix | CPU mix |
| Apple 96 GB+ unified | Yes | Yes | Yes | Yes | Yes | Tight |
| System RAM only (32 GB+) | Yes | Yes | CPU mix | CPU mix | CPU mix | CPU mix |
The pattern is clear: a single 24 GB card tops out at a painful Q2_K, and a useful Q4_K_M needs either two 24 GB cards or a 64 GB unified-memory Mac. That is the real answer to “can I run 70B” for most people.
The two practical paths
Two consumer GPUs. Two 24 GB cards give roughly 48 GB of addressable VRAM after driver reserve. Splitting the 70B layers across both lets Q4_K_M and even Q5_K_M run with speed close to a single large card. This is the most common enthusiast setup because the cards are cheap on the used market.
Unified memory. An Apple Silicon Mac with 64 GB or more exposes that memory to the model as one pool. A 70B at Q4_K_M needs about 42 GB all in, which leaves room on a 64 GB machine and comfortable headroom on 96 GB. The catch is that macOS keeps part of the memory, so read the Apple Silicon page before assuming the full number is yours.
When CPU offload is the only option
If you have one 24 GB card and insist on 70B, the engine will offload the layers that do not fit into system RAM. Generation still works, but every token pays the cost of shuttling weights between RAM and VRAM, so output slows to a crawl. This is fine for a one-off summarization job run overnight and miserable for chat. The layer split is controlled by your engine’s offload setting, which trades speed for fit.
Why you might not need 70B
A 70B at Q4_K_M behaves well, but a 13B at Q6_K or Q8_0 often matches it on narrow tasks and fits on a single 16–24 GB card. Before committing to the multi-GPU or unified-memory expense, ask whether a smaller model at a higher tier serves you. The quantization comparison shows how much quality each tier keeps.
Estimating your own case
You do not have to memorize the matrix. Enter your model’s parameters, the tier’s bits, its layer and head counts, and your available memory into a VRAM calculator. If the total beats your card by a few gigabytes, that gap is exactly what a second GPU or a larger unified-memory tier closes. If it beats your card by tens of gigabytes, drop a quant tier or pick a smaller model.
The five paths, stated plainly
The hardware matrix above breaks down by quant tier. This view flips it: five realistic paths, each judged on whether it runs 70B at all, what it needs, and what it costs. The magnitudes are planning estimates that depend on the exact quant, engine, and overhead.
| Path | Can it run 70B? | What it needs | Main cost |
|---|---|---|---|
| Single 24 GB card | Barely, at Q2_K only | Most layers offloaded to system RAM; almost no headroom for cache or engine | Extremely slow generation; no room for long context or batches |
| Two 24 GB cards (layer split) | Yes through Q4_K_M, even Q5_K_M | Two matching cards and an engine that splits layers across them | Some interconnect overhead; the used-market hunt for cheap cards |
| Single 48 GB card | Yes through Q4_K_M and Q5_K_M | A workstation-grade 48 GB GPU | Higher card cost than two consumer cards; one device to fail |
| 64–96 GB unified memory (Apple Silicon) | Yes through Q4_K_M; 96 GB reaches Q6_K | A Mac with 64 GB or more; macOS keeps part of the memory | Lower peak bandwidth than a discrete GPU; capped by iogpu.wired_limit_mb |
| CPU offload / system RAM | Fits at most quants | 32 GB or more of system RAM and any GPU for the offloaded slice | Largest speed loss; per-token fetch from RAM — fine for batch, poor for chat |
The single 24 GB card is the trap: Q2_K weights are about 22.8 GB (from the quant table), so on a 24 GB card the weights alone leave roughly 1 GB, which the KV cache and engine immediately consume. The realistic floor is therefore two 24 GB cards, a 48 GB card, or a 64 GB unified-memory machine. The multi-GPU VRAM page explains how the split itself behaves once you take the two-card path.
Can pressing the quant harder save it?
Dropping a quant tier trades quality for weight space, and the gap is large at the low end. From Q4_K_M (about 39.4 GB) to Q3_K_M (about 30.6 GB) frees roughly 9 GB; to Q2_K (about 22.8 GB) frees roughly 16.6 GB. That is enough to pull a 70B onto a single 24 GB card at Q2_K, but only just — the ~1 GB of remaining room leaves no space for the KV cache or engine overhead, so the run is fragile.
The direction matters more than the exact figure: each step down the quant ladder buys weight space and pays it back in quality. At Q2 and Q3 the quality loss is severe (the quantization quality loss page covers the direction), and the memory margin stays thin enough that any long context or small batch tips the run into OOM. Pressing to Q2/Q3 is a rescue for “I must load this once,” not a setup to rely on. Where the budget allows, a second card or a larger unified-memory tier is the calmer answer.
What you actually run into
The static fit number is only the first question. A few things surface only once the model is running, and all are qualitative.
- Context keeps growing. A session that loaded fine at 4K can hit the cache ceiling mid-conversation as the history accumulates. The weights did not move; the cache did, exactly as the context length cost page describes.
- Concurrency multiplies the cache. Serving more than one user at a time keeps one cache per live sequence on top of the shared weights, so a setup with comfortable single-user headroom can fall over under a few simultaneous chats.
- Load time and offload latency. Larger quants and offloaded layers take longer to load, and a CPU-offloaded slice is fetched every token, which is felt as slower replies rather than as a memory error. None of this changes the fit verdict, but it changes whether the run is pleasant to use.
Plan for the peak, not the average: the longest context you will actually hold and the most users you will actually serve, then check that against the fit.
A decision checklist
Start from what you want the model to do, then work back to a path.
- Interactive chat on a tight budget. Two 24 GB cards at Q4_K_M, or a 64 GB unified-memory Mac. Both give a usable 70B without offload pain.
- Highest quality from one model. A 48 GB card or a 96 GB unified machine, reaching Q5_K_M or Q6_K, where the weights leave real room for cache.
- Only one 24 GB card available. Do not force 70B; a 13B or 34B model at a high quant fits with margin and often matches 70B on narrow tasks.
- No GPU, offline batch work. CPU offload into system RAM. It fits, but replies are slow, so reserve it for jobs that can wait.
- Long context is the priority. Budget the cache headroom first and prefer the larger memory pool, because the cache — not the weights — is what long context spends.
If the goal is “a capable local model for daily chat,” the 70B question is often answered by a smaller model at a higher quant on the card you already own; if the goal is “the biggest model I can,” the path is the two-card or unified-memory route above. The VRAM calculator turns any of these choices into a concrete total.