Context Length Costs VRAM: Sizing a Real Session

How context length drives KV cache VRAM: a chat-length to KV-cache table plus how batching and concurrency multiply memory on top of the model weights.

Context Length Costs VRAM: Sizing a Real Session

Most out-of-memory errors at load time are about weights. Most out-of-memory errors during a session are about context. This page shows how chat length turns into KV cache memory and how running several conversations at once multiplies that cost on top of the model itself.

The linear rule

The KV cache holds one key and one value vector for every token in the context. Because context length is a straight multiplier in the cache formula, the relationship is linear: double the tokens, double the cache. The weights stay fixed; only the cache moves.

This is the single most useful fact on the page. If a model loads at 4k context and fails at 32k, the weights did not change. The cache did, fourfold.

Chat length to KV cache, worked

Using a representative shape of 40 layers, 8 KV heads, 128 head dimension, and a 16-bit cache (2 bytes per element), the per-token cache is:

2 × 40 × 8 × 128 × 2 = 163,840 bytes ≈ 0.00015 GB per token

Scale that by context length:

At 128k context the cache alone is larger than many graphics cards. A model that fits easily at 4k can become impossible at 128k purely from the cache, with zero change to the weights.

Batching multiplies the cache

A single chat uses one KV cache. A server handling several users at once keeps one cache per active sequence. If four people each hold an 8k context, the engine stores four caches:

4 × 1.34 GB ≈ 5.37 GB of KV cache on top of the shared weights.

The weights are loaded once and shared; the caches are not. Concurrency is therefore the silent memory multiplier that bites self-hosted servers. Plan for peak simultaneous contexts, not average, because the cache memory is allocated per live sequence.

Sizing a real session

1. Estimate your longest real exchange in tokens, not characters. A few thousand words is roughly a few thousand tokens, but code and repeated text vary.
2. Compute the KV cache for that length with the formula above.
3. Add it to the weight size from the quantization comparison.
4. Add engine headroom and, if serving users, multiply the cache by expected concurrent sessions.
5. Check the sum against your card’s usable memory.

Practical defaults

• For interactive chat, 4k–8k context covers most exchanges and keeps the cache under about 1.5 GB on the shape above.
• For document question answering, 16k–32k is common; budget several gigabytes accordingly.
• For “ingest a whole book” use, 128k is tempting but the cache alone can exceed a 24 GB card. Prefer chunking or a larger memory pool on a Mac with high unified memory.

Estimating your prompt in tokens

You rarely know the exact token count until you run it. A few rules of thumb help you size context without a tokenizer open:

1. English prose runs about 0.75 token per word, so 1,000 words is roughly 750 tokens.
2. Code and JSON are denser, often 1.5 to 2 tokens per word, because symbols each become their own tokens.
3. Repeated boilerplate, such as a system prompt pasted into every call, counts every time and quietly eats the budget.
4. A long document you paste for question answering is the largest single source; count it, do not guess it.

If your estimate plus the model’s fixed overhead approaches the context you set, lower the setting rather than risk an OOM mid-reply.

The trap of maximum context

Model cards advertise a large max context, but that is a capability, not a free setting. Enabling it reserves cache memory whether or not you use it. Set the context to what your session needs and no more; the savings are immediate and often the difference between a stable run and an OOM mid-conversation.