KV Cache Size Explained: The Formula Behind Context Length

Why long context eats VRAM: the KV cache formula, how GQA shrinks it, and a worked table showing KV cache doubling when you double the context length.

KV Cache Size Explained: The Formula Behind Context Length

When a model runs out of memory at a long context but handled a short one fine, the KV cache is usually the reason. This page shows the formula, why context length is the dominant lever, and how modern attention designs quietly shrink the cache.

The core formula

The KV cache size in bytes is:

KV cache (bytes) = 2 × layers × kv_heads × head_dim × sequence_length × bytes_per_element

The leading 2 covers the separate K (key) and V (value) tensors. The product kv_heads × head_dim is the size of one key or value vector. sequence_length is the number of tokens held in context, and bytes_per_element is the precision the cache is stored at.

You can also write it per token, which drops the sequence length:

KV cache per token (bytes) = 2 × layers × kv_heads × head_dim × bytes_per_element

Multiply that by the context length to get the full cache. This per-token form makes the linear relationship obvious: every extra token adds the same fixed amount.

The “doubling” property

Because sequence length is a straight multiplier, halving or doubling the context halves or doubles the KV cache. The weights do not change when you change context; only the cache does. That is why a model that fits at 4k context can fall over at 32k even though the file on disk is identical.

The table below works the per-token number for three real attention shapes, then shows the full cache at four context lengths. All use 16-bit (2 bytes per element) for the cache.

Read the rows left to right: each step doubles the context and doubles the cache. The small MQA model stays tiny because it has a single KV head; the GQA rows show how a modest head count still produces multi-gigabyte caches at long context.

Where GQA actually helps

Grouped Query Attention was adopted widely because it attacks exactly the kv_heads term. A model with 32 attention heads but only 8 KV heads stores one quarter as many key and value vectors per token as a full multi-head design. The quality stays close because the query heads still read from the shared KV heads.

Multi-Query Attention takes this further with a single KV head, which is why the MQA row above is so small. The trade is sometimes a small hit to quality on certain tasks, but the memory saving at long context is large.

You can see the exact KV head count in a model’s config (often num_key_value_heads in GGUF or safetensors metadata). Plug it into the VRAM calculator to get a per-model number rather than a rule of thumb.

KV cache versus weight memory

It is easy to assume the weights dominate. For short context they usually do. At long context the cache can catch up or pass them, especially on small models with many layers. Consider a 7B model at 4-bit: weights near 4 GB. At 4k context the cache might be a few hundred megabytes. At 128k context the cache can reach several gigabytes and rival the weights.

This is the practical warning behind the context length cost page: the budget you set for context is not a rounding error, it is a first-class memory line.

Precision of the cache itself

The formula above assumed 16-bit elements for the cache. Some engines store the KV cache at the model weight precision, others at 16-bit regardless, and a few support a dedicated KV quant. Because the cache can be large, the choice matters: dropping the cache from 16-bit to 8-bit roughly halves that term. Check your engine’s settings; the quantization comparison covers what each file format actually stores.

Why attention cost is a different curve

Do not conflate the two bottlenecks. The KV cache grows linearly with context. The attention computation, the matrix multiplies over the full sequence each step, grows roughly with the square of the context. Long context therefore costs you in two ways at once: more stored cache and more compute per token. The memory part is what decides whether the model loads; the compute part is what decides how slow it feels.

A quick check you can do by hand

Pick any model. Find layers, KV heads, and head dim from its config. Multiply 2 × layers × kv_heads × head_dim, then by 2 bytes if the cache is 16-bit. That is the per-token cache. Multiply by your planned context and divide by a billion for gigabytes. If the number surprises you, that surprise is the whole point of this page.