Ollama Out of Memory: What Actually Frees VRAM
Ollama out-of-memory fixes that work: the real causes, what num_gpu and num_ctx do, and a symptom-to-root-cause table you can act on.
Ollama Out of Memory: What Actually Frees VRAM
Ollama reports out of memory when a load or a generation asks for more VRAM than is free. The fix is almost never a reinstall; it is matching the model, context, and offload settings to the card you have. This page gives a symptom table and explains the two settings that matter most.
The two settings that control memory
Ollama exposes its memory behavior through model options. The two with the largest effect are:
num_gpu— the number of layers kept on the GPU. Each layer left on the GPU uses VRAM but runs fast; each layer moved to CPU uses system RAM instead and runs slow. Raisingnum_gputo the maximum your card allows is fastest; lowering it trades speed for fit.num_ctx— the context window in tokens. The KV cache scales linearly with this number, so a highnum_ctxcan quietly consume several gigabytes. This is the setting people forget.
A typical invocation looks like:
ollama run llama3:70b-instruct-q4_K_M -o num_gpu=48 -o num_ctx=4096If it fails, drop num_ctx before you drop num_gpu, because context is usually the cheaper thing to lose.
Symptom to root cause table
| Symptom | Likely root cause | What to do |
|---|---|---|
| Model refuses to load at all | Weights exceed free VRAM | Lower num_gpu, or use a lower quant tier from the quantization comparison |
| Loads, then OOM on first long reply | KV cache for large num_ctx too big | Reduce num_ctx to shrink the cache |
| Worked yesterday, fails today | Stale VRAM from a prior run or another process | Restart Ollama; close other GPU apps |
| OOM only with multiple concurrent chats | Batching multiplies KV cache | Lower num_ctx or limit concurrency |
| Fails on a 24 GB card at Q4 | Model simply too large for one card | Offload more layers, use two GPUs, or go smaller model |
| Slow but no OOM after lowering num_gpu | Too many layers on CPU | Raise num_gpu until it just fits |
Steps that reliably free memory
- Restart the Ollama service. This releases VRAM held by a previous load that did not clean up.
- Lower
num_ctx. Cutting context from 32768 to 4096 can free several gigabytes on its own. - Lower
num_gpugradually. Each step moves layers to system RAM, trading speed for fit. - Pick a lower quant tier. Moving a 70B from Q4_K_M to Q3_K_M drops the weights by roughly a quarter.
- Close other GPU users. A browser with hardware acceleration or a second model server can hold gigabytes you need.
- Verify with a memory read using your system’s monitoring tools to see what is actually allocated.
Why the model sometimes loads and sometimes not
A model near the limit is unstable by definition. Between runs, the driver may not have released every allocation, or a background process grabbed part of the card. The same file that loaded cleanly at 10 AM can throw OOM at 10:05 with a slightly larger context. Treat any model that sits within a gigabyte of your card as unreliable under load, and either offload more or downsize. A clean restart between attempts is the single most reliable fix for a borderline model.
Offload is the same idea as llama.cpp
Ollama’s num_gpu and llama.cpp’s -ngl are the same mechanism: keep the first N layers on the GPU and compute the rest on CPU. The offload layers guide explains how the split changes speed and memory in detail. If you tune one, you understand the other.
A quick triage order
When you hit OOM, do not guess. Restart, then re-run with num_ctx=2048. If it loads, raise context until it stops. If it still fails, lower num_gpu. If it still fails, the weights themselves are too big and the only real fixes are a smaller model, a lower quant, or more total memory as described on the 70B local page.
Frequently asked questions
What does Ollama's 'out of memory' error actually mean?
It means the request needed more VRAM than was free at that moment. The cause is usually the model size plus the KV cache for the requested context, not a code bug. Reducing either frees the memory.
What does the num_gpu setting control?
It sets how many layers Ollama keeps on the GPU. A higher number keeps more compute on the GPU and faster; a lower number pushes layers into system RAM, which saves VRAM but slows generation.
What does num_ctx do to memory?
It sets the context window size. Because the KV cache scales with context length, a large num_ctx can consume gigabytes on its own. Lowering it is often the fastest fix for an OOM.
Why does the same model load sometimes and fail other times?
Leftover allocations from a previous run, another process using the GPU, or a larger context on the new request can tip a borderline model over the limit. Freeing the GPU and shrinking context restores it.
Does restarting Ollama help?
Often yes, because it releases stale VRAM held by a previous load. After restart, set a smaller num_ctx or lower num_gpu so the model stays within your card's real limit.