llama.cpp GPU Offload Layers (-ngl) Explained

What llama.cpp -ngl does: how many layers to offload to your GPU, the speed and memory tradeoff from -ngl 0 to all layers, and copy-paste command examples.

llama.cpp GPU Offload Layers (-ngl) Explained

The -ngl flag (short for n-gpu-layers) is the main lever for fitting a GGUF model on your hardware in llama.cpp and the tools built on it. This page explains what it does, how the speed and memory change as you sweep it from zero to all layers, and shows commands you can copy.

What offloading means

A transformer model is a stack of layers. -ngl N keeps the first N layers on the GPU and runs the remaining layers on the CPU. The GPU layers use VRAM; the CPU layers use system RAM. The weights for the whole model are still read from disk into RAM, but only the GPU layers are copied into VRAM for compute.

This is the same idea as Ollama’s num_gpu. If you have tuned one, you understand the other; the Ollama OOM guide covers the equivalent knobs there.

How memory and speed move with -ngl

The table is qualitative because exact numbers depend on the model and drivers, but the shape is consistent. “VRAM” is the GPU memory used; “speed” is relative token throughput.

The key insight is the cliff between “most” and “all”. A model with most layers offloaded still shuffles data to the CPU every token, so it is much slower than the same model with every layer on the GPU. The goal is therefore to offload as many layers as fit, not merely “enough to load”.

Finding the layer count

You need the model’s total layer count to know what “all” means. It is in the GGUF metadata (often n_layers) and printed by llama.cpp at load time. A 7B model commonly has 32 layers; a 70B model commonly has 80. Enter those into the VRAM calculator to estimate the VRAM per layer.

Command examples

Load a 7B model with all layers on the GPU, 4096 context:

./llama-cli -m model.Q4_K_M.gguf -ngl 32 -c 4096

Let the engine decide how many layers fit, by requesting more than exist:

./llama-cli -m model.Q4_K_M.gguf -ngl 999 -c 4096

Run mostly on CPU with a little GPU help on a small card:

./llama-cli -m model.Q4_K_M.gguf -ngl 10 -c 2048

For a 70B model on two GPUs, set the split and offload all layers:

./llama-cli -m model.Q4_K_M.gguf -ngl 80 -sm row -c 4096

The -sm row flag spreads the model across GPUs row-wise; exact behavior depends on your build, so confirm with a memory read.

Tuning procedure

1. Start at -ngl 999 and let llama.cpp take what fits. Note the layer count it actually offloads.
2. If it fails, lower -ngl in steps of 4 until the load succeeds.
3. Watch VRAM with your system monitor to see the headroom.
4. Once stable, raise -c (context) only if memory allows; context eats KV cache as explained on the context length page.

Common mistakes

• Setting -ngl to exactly the layer count but forgetting the KV cache, which pushes the card over its limit. Leave a gigabyte or two free.
• Assuming partial offload is “almost as fast” as full. It is not, because of the per-token memory transfers.
• Forgetting to re-measure after a driver update. A layer count that fit last week can overflow once an engine upgrade changes its buffer size.
• Raising context after a successful load and hitting OOM on the next request, because the KV cache grew.

Offload is a budget you spend between speed and fit. Spend it on layers first, then on context, and keep a small reserve for the engine itself.