Mixture-of-Experts (MoE) models split their feed-forward layers into many expert sub-networks and activate only a few per token. That design makes them efficient to compute, but it creates a persistent confusion about memory: people size the model by its small active count and are surprised when it will not load. This page separates the two numbers and shows which one your VRAM bill follows.
Total parameters decide memory
An MoE model must keep every expert’s weights in memory so the router can choose among them. The router picks a handful of experts per token, but the unchosen experts are still resident. Therefore the VRAM requirement scales with total parameters, not active ones.
The active parameter count governs how much math happens per token, which is a speed question. The total count governs how much memory the file needs, which is a fit question. They are different axes and must not be swapped.
Total versus active, worked
The table uses representative ratios. Real models vary, but the gap between total and active is the point.
| Model type | Total params | Active per token | Weight memory at Q4_K_M (approx) | Compute per token |
|---|---|---|---|---|
| Dense 13B | 13 B | 13 B | 7.3 GB | Full 13B |
| MoE 13B-class | 13 B | ~2 B | 7.3 GB | ~2 B |
| Dense 70B | 70 B | 70 B | 39.4 GB | Full 70B |
| MoE 70B-class | 70 B | ~10 B | 39.4 GB | ~10 B |
| MoE 140B-class | 140 B | ~20 B | 78.8 GB | ~20 B |
The weight memory column is identical between a dense model and an MoE of the same total size, because both load all weights. The compute column is where the MoE wins: far less work per token for the same memory footprint.
MoE versus a dense model of equal total
Compare an MoE with 70B total and ~10B active to a dense 70B:
- Memory: the same. Both need about 39 GB at Q4_K_M. Neither fits a single 24 GB card.
- Speed: the MoE is faster per token because it activates ~10B, not 70B, of compute.
- Fit decision: identical. The hardware guidance for a dense 70B applies to both, because the budget is the total count.
This is why the headline “only 10B active” is misleading for memory planning. It tells you the model will feel quick, not that it will load on a small card.
Sizing an MoE for your machine
Treat the total parameter count exactly as you would a dense model:
- Take total parameters (ignore active for memory).
- Multiply by the quant bits and divide by 8 for weight size, as on the quantization comparison.
- Add the KV cache for your context, which scales with the sequence length.
- Add engine headroom.
- Compare to your card’s usable memory from the GPU reference.
A calculator that takes total parameters will give an MoE and a dense model of the same total the same memory estimate. That is the correct result.
Where active parameters still matter
Do not throw the active count away; it sets your speed and your power draw. If two models have similar total size but very different active counts, the one with fewer active params will generate faster on the same hardware. It just will not fit in less memory. Knowing both numbers lets you pick a model that is simultaneously loadable and quick, rather than optimizing for one and breaking the other.
Reading an MoE spec
One practical consequence: when you see a MoE advertised by its active parameter count, mentally multiply up to the total before judging fit. A model quoted at “just 12B active” with 140B total needs the memory of a 140B dense model and will not load where a real 12B would. The active number is a speed headline, not a memory one.
The one-line rule
Memory follows total parameters; speed follows active parameters. Size for the total, expect speed from the active, and you will never be caught out by an MoE that “should have fit” because its active count looked small.
Dense versus MoE at the same total size
The worked table above shows the numbers; this one shows which number to read for which decision. At equal total parameter count a dense model and an MoE are twins for memory and strangers for speed.
| Question you are answering | Dense model | MoE of equal total | Read this number |
|---|---|---|---|
| How much VRAM will it need? | Total params | Total params (identical) | Total parameter count |
| How fast will it generate? | All params per token | Active params per token (fewer) | Active parameter count |
| When is the MoE the better buy? | — | When the memory fits but you want quicker generation | Compare active against total |
| When does dense win on simplicity? | One number to reason about | Two numbers to track | The active-to-total ratio |
The MoE is the better fit when the hardware can hold the full total but the workload benefits from faster per-token compute — long contexts, many turns, or agent loops where the cheaper active path adds up. It is not better when the total alone will not load, because the memory bill is the same. The active count only changes the speed expectation, so a small active number next to a large total is a speed headline, not a memory reprieve. The LLM engine comparison page covers how runtimes handle the heavier total, and the quantization comparison shows how the weight tier still scales the whole total.
Can experts be offloaded or loaded partially
Offloading an MoE works the same way as offloading a dense model: moving layers, including expert layers, from the GPU into system RAM frees VRAM and slows generation. The trade is the same one described on the CPU offload performance page — the savings are roughly the weight size of what moved, and the cost is paid per token through the bandwidth gap.
Loading fewer experts than the full set is a different idea and is generally not a usable lever. The router chooses which experts to engage on each token, and it can only choose among experts that are resident in memory. That means all experts must be loaded for the router to function, so the total stays resident regardless of how many run per token. The active count is a compute path, not a loading shortcut. The practical options therefore remain the standard ones: offload to save VRAM at a speed cost, or choose a smaller total. There is no standard setting that keeps the memory of a small model while holding the weights of a large one.
How to read active parameters on a model card
The two numbers an MoE card quotes map to two config fields. To read them:
- Total experts appears as
num_experts(orn_routed_experts) in the model’sconfig.json. This sets how many expert blocks exist. - Experts used per token appears as
num_experts_per_tok(orexperts_per_tok/moe_topk). This is how many of those blocks the router activates on each token. - The active parameter count is approximately the total scaled by the ratio of activated experts to total experts for the feed-forward portion. It is an estimate that depends on the exact architecture, so treat it as a direction, not a precise figure.
The common misread is to take the active number as the memory number. A model advertised at “just 12B active” with 140B total needs the memory of a 140B dense model and will not load where a real 12B would. The active figure is the speed headline; the total is the loading figure. Always read both before judging fit, and the model file size lookup shows the on-disk weight size that the total implies for each quant tier.
Worked estimate for an MoE-shaped model
The following is an estimate built only from the site’s shared figures, to show how the two numbers pull in different directions. Take a 70B-class MoE: about 70B total parameters with roughly 10B active per token, matching the representative ratios in the site’s model reference.
- VRAM follows the total. At Q4_K_M the weight block is estimated at about 39.4 GB, from the site’s quant table (70B × 4.5 bits ÷ 8). This is the same weight memory a dense 70B would need, because both load every parameter.
- KV cache is separate. Using a 70B-class attention shape of 80 layers, 8 KV heads, and 128 head dimension, the per-token KV cost is about 320 KB, and at a 32K context the cache is estimated at about 10 GB. This depends on the context length and is held at 16-bit regardless of the weight quant.
- Total estimate. Adding headroom, the model is estimated at roughly 50 GB of VRAM at 32K context. The exact figure depends on the chosen context length, the quant tier, and the engine’s reserve, so treat it as an estimate, not a measured reading.
- Speed follows the active. Generation is driven by the roughly 10B active parameters per token, so the per-token compute is closer to a 10B-class model than to a 70B dense, even though the memory equals the 70B dense.
The point of the estimate is the split: the memory line is set by the total and the speed line by the active, and the two should never be swapped. The VRAM calculator reproduces this estimate when given the total, and the KV cache calculator reproduces the cache term when given the shape and context.