MoE Models and VRAM: Total vs Active Parameters
Why MoE VRAM depends on total parameters, not active ones, and how that differs from a dense model of the same size when fitting locally.
MoE Models and VRAM: Total vs Active Parameters
Mixture-of-Experts (MoE) models split their feed-forward layers into many expert sub-networks and activate only a few per token. That design makes them efficient to compute, but it creates a persistent confusion about memory: people size the model by its small active count and are surprised when it will not load. This page separates the two numbers and shows which one your VRAM bill follows.
Total parameters decide memory
An MoE model must keep every expert’s weights in memory so the router can choose among them. The router picks a handful of experts per token, but the unchosen experts are still resident. Therefore the VRAM requirement scales with total parameters, not active ones.
The active parameter count governs how much math happens per token, which is a speed question. The total count governs how much memory the file needs, which is a fit question. They are different axes and must not be swapped.
Total versus active, worked
The table uses representative ratios. Real models vary, but the gap between total and active is the point.
| Model type | Total params | Active per token | Weight memory at Q4_K_M (approx) | Compute per token |
|---|---|---|---|---|
| Dense 13B | 13 B | 13 B | 7.3 GB | Full 13B |
| MoE 13B-class | 13 B | ~2 B | 7.3 GB | ~2 B |
| Dense 70B | 70 B | 70 B | 39.4 GB | Full 70B |
| MoE 70B-class | 70 B | ~10 B | 39.4 GB | ~10 B |
| MoE 140B-class | 140 B | ~20 B | 78.8 GB | ~20 B |
The weight memory column is identical between a dense model and an MoE of the same total size, because both load all weights. The compute column is where the MoE wins: far less work per token for the same memory footprint.
MoE versus a dense model of equal total
Compare an MoE with 70B total and ~10B active to a dense 70B:
- Memory: the same. Both need about 39 GB at Q4_K_M. Neither fits a single 24 GB card.
- Speed: the MoE is faster per token because it activates ~10B, not 70B, of compute.
- Fit decision: identical. The hardware guidance for a dense 70B applies to both, because the budget is the total count.
This is why the headline “only 10B active” is misleading for memory planning. It tells you the model will feel quick, not that it will load on a small card.
Sizing an MoE for your machine
Treat the total parameter count exactly as you would a dense model:
- Take total parameters (ignore active for memory).
- Multiply by the quant bits and divide by 8 for weight size, as on the quantization comparison.
- Add the KV cache for your context, which scales with the sequence length.
- Add engine headroom.
- Compare to your card’s usable memory from the GPU reference.
A calculator that takes total parameters will give an MoE and a dense model of the same total the same memory estimate. That is the correct result.
Where active parameters still matter
Do not throw the active count away; it sets your speed and your power draw. If two models have similar total size but very different active counts, the one with fewer active params will generate faster on the same hardware. It just will not fit in less memory. Knowing both numbers lets you pick a model that is simultaneously loadable and quick, rather than optimizing for one and breaking the other.
Reading an MoE spec
One practical consequence: when you see a MoE advertised by its active parameter count, mentally multiply up to the total before judging fit. A model quoted at “just 12B active” with 140B total needs the memory of a 140B dense model and will not load where a real 12B would. The active number is a speed headline, not a memory one.
The one-line rule
Memory follows total parameters; speed follows active parameters. Size for the total, expect speed from the active, and you will never be caught out by an MoE that “should have fit” because its active count looked small.
Frequently asked questions
Does an MoE model need VRAM for all experts or just the active ones?
It needs VRAM for all of them. Every expert's weights must be resident to pick from, so memory scales with total parameters even though only a subset runs per token.
What do active parameters control then?
Active parameters set the compute per token, which drives speed. A model with many total but few active parameters is memory-heavy but can still be fast, the opposite of a dense model.
Is an MoE cheaper to run than a dense model with the same total size?
Memory is about the same because both must hold all weights. The MoE is usually faster per token because it activates fewer parameters, but it will not fit in less VRAM.
How do I size an MoE for my card?
Use the total parameter count with the target quant in the [VRAM calculator](/vram-calculator/), exactly as you would for a dense model. The active count only changes your speed expectation.
Why do guides get this wrong?
They conflate the active parameter count, which is small and quoted in headlines, with the total, which is what you load. The two differ by several times on big MoE models.