LLM VRAM Math Cheatsheet

Every VRAM formula for local LLMs on one page: weight size, KV cache, engine overhead, batch multipliers, and worked conversions you can do by hand.

LLM VRAM Math Cheatsheet

This page collects the memory formulas for running models locally in one place. Use it to check a calculator result or to estimate a model on paper. Every line is hand-computable from the model’s published shape.

Weight memory

weight_GB = parameters_billions × quant_bits ÷ 8

The ÷ 8 converts bits per parameter to bytes per parameter. Examples:

• 7B at 4 bits: 7 × 4 ÷ 8 = 3.5 GB
• 7B at 8.5 bits (Q8_0): 7 × 8.5 ÷ 8 = 7.44 GB
• 70B at 4.5 bits (Q4_K_M): 70 × 4.5 ÷ 8 = 39.38 GB

This is the floor. The model will not use less than this, and usually uses more once the cache and engine join.

KV cache

kv_bytes = 2 × layers × kv_heads × head_dim × context × bytes_per_element
kv_GB   = kv_bytes ÷ 1,000,000,000

The leading 2 is the K and V tensors. Per token, drop the context term:

kv_per_token_bytes = 2 × layers × kv_heads × head_dim × bytes_per_element

At 16-bit cache (2 bytes per element) on a 40-layer, 8-KV-head, 128-head-dim model:

2 × 40 × 8 × 128 × 2 = 163,840 bytes per token

Times 32,768 context: 5.37 GB. The context length cost page builds a full ladder from this.

GQA effect

Because kv_heads sits in the formula, cutting it cuts the cache directly. A model with 8 KV heads instead of 32 stores one quarter the KV cache per token. The worked comparison above shows this effect.

Engine and headroom

There is no exact formula; the engine needs memory for the compute context, graphs, and buffers, plus a reserve so the model is not at the limit. A planning estimate of a few gigabytes is typical, scaling with batch size and engine. The GPU reference explains why usable memory is below nominal.

Total memory

total_GB = weight_GB + kv_GB + engine_headroom_GB

Compare total_GB to your card’s usable memory, not its nominal memory.

Batch and concurrency multiplier

kv_total_GB = kv_per_sequence_GB × concurrent_sequences

Weights are shared; caches are not. Serving four chats multiplies the KV term by four. Activations also grow with batch size, adding more on top.

MoE adjustment

For mixture-of-experts models, use the total parameter count in the weight formula. Active parameters change speed, not memory.

Quick conversion reminders

• 1 byte per parameter at 8 bits; 0.5 byte at 4 bits.
• 1 GB = 1,000,000,000 bytes for these estimates (decimal, not binary).
• To go from bits per parameter to bytes, divide by 8.
• To go from GB back to parameters: parameters_billions = weight_GB × 8 ÷ quant_bits.
• To sanity-check, recompute the same model at two quants; the difference should match the bit delta times the parameter count.

Worked end-to-end example

Take a 34B model at Q4_K_M (about 4.5 bits), 60 layers, 8 KV heads, 128 head dimension, 16-bit KV cache, and an 8k context on a 24 GB card.

1. Weight = 34 × 4.5 ÷ 8 = 19.13 GB.
2. KV per token = 2 × 60 × 8 × 128 × 2 = 245,760 bytes.
3. KV at 8k = 245,760 × 8,192 ÷ 1,000,000,000 = 2.01 GB.
4. Engine headroom = 1.5 GB (estimate).
5. Total = 19.13 + 2.01 + 1.5 = 22.64 GB.
6. Usable on a 24 GB card after the reserve is about 21 GB, so this does NOT fit comfortably; drop to Q3_K_M or lower context.

Running the same numbers in a calculator should return the same total, which is the check that your hand math is right.

A two-minute estimate

Given a model’s parameters, quant tier, layers, KV heads, head dim, and your context:

1. Weight = params × bits ÷ 8.
2. KV = 2 × layers × kv_heads × head_dim × context × bytes ÷ 1e9.
3. Add a few GB headroom.
4. Check against usable memory.

If that fits with margin, load it and confirm with a live memory reading. The math here is the plan; the live reading is the proof.