How to Measure Real VRAM Usage While Running a Model

Three commands to measure actual VRAM while a model runs on NVIDIA, macOS, and Linux, plus a table decoding what each reading means for your fit decision.

How to Measure Real VRAM Usage While Running a Model

Estimates tell you whether a model should fit; measurement tells you whether it actually does. This page gives one concrete command per platform and a table for reading what the output means, so you can confirm a fit before trusting it.

NVIDIA: nvidia-smi

While the model is loaded, run:

nvidia-smi

The output lists each GPU with Memory-Usage as used / total in MiB. For a live view during generation, watch it:

watch -n 1 nvidia-smi

Read the used value at the moment of peak context. That is the true memory the model plus engine consumed, including the KV cache it built.

macOS: vm_stat and Activity Monitor

Apple Silicon has no VRAM counter because memory is unified. Check the pool with:

vm_stat

It prints page counts; Pages free, Pages inactive, and Pages occupied by compressor tell you how tight the pool is. For a model pushing past physical memory, watch Pages swapped out climb, which means the system is swapping and generation will stall. Activity Monitor’s Memory tab shows the same in gigabytes and is easier to read at a glance.

Linux without a GPU: free and /proc/meminfo

Under CPU offload the model lives in system RAM. Check it with:

free -h

or watch the resident set of the process:

watch -n 1 "cat /proc/meminfo | head -3"

The Mem line shows used versus total RAM. A model loaded into RAM appears as a large chunk of used memory; if available drops near zero, the machine will swap and slow drastically.

What each reading means

Turning measurement into a decision

1. Load the model at your normal context.
2. Note the used memory at load. Compare it to the VRAM calculator estimate; a large gap means the engine holds more than you modeled.
3. Send a long prompt and watch the number rise. The increase is the KV cache, which grows with the context.
4. If used approaches total, step down a quant tier from the quantization comparison or lower context.
5. On Apple Silicon, if swapping begins, the model is over the real usable pool, and you should step down a tier.

A worked reading

Suppose nvidia-smi shows a 24 GB card with 21.4 GB used after loading a 13B model at Q4_K_M with an 8k context. The calculator estimated about 7.3 GB of weights plus roughly 1.3 GB of KV cache plus a couple of gigabytes of overhead, which lands near the observed figure. If instead the reading shows 23.8 GB used at load, the model is at the edge and the first long reply will likely fail; step down a quant tier or lower context before trusting it. The live number is the verdict, not the estimate.

Why measure at peak, not load

A model can load inside the limit and then exceed it once the KV cache fills during a long reply. The load reading is the floor, not the ceiling. Always measure during the longest exchange you actually intend to run; that peak is the number your hardware must survive.

When the numbers disagree with the estimate

If measurement shows far more memory than the calculator predicted, the engine is holding buffers or the KV cache is at higher precision than assumed. If measurement shows less, you may have left context headroom unused and can raise it. Either way, trust the live number for the final fit decision and keep the GPU reference handy for comparing against your card’s nominal total.