VRAM Calculator: How Much GPU Memory Your LLM Needs
Short answer: VRAM = (parameters x bytes per weight) + KV cache + about 1 GB of runtime overhead. At Q4_K_M — roughly 0.56 bytes per weight — that is close to 0.6 GB per billion parameters, so a 7B model needs about 5.5 GB, a 32B about 20 GB and a 70B about 42 GB at a 4K context. Double those weight figures for FP16.
Set the sliders below for your own model, quantization and context length. The numbers apply to Ollama, llama.cpp, LM Studio and vLLM alike, because they all load the same weights.
Calculate Your VRAM Needs
Good — minimal quality loss (recommended)
✅ Model Fits in Your GPU!
How is VRAM calculated for an LLM?
Three things occupy the card, and only the first one is obvious.
1. Model weights
The largest term: parameters x bytes per weight. An 8B model at FP16 is 8 x 2 = 16 GB. The same model at Q4_K_M is 8 x 0.5625 = 4.5 GB. Quantization is the only lever that moves this number.
2. KV cache
Attention keys and values for every token in the window, so it grows linearly with context — and it does not shrink when you quantize the weights. Its size depends on layer count and grouped-query attention, not on parameter count. See the per-model KV table below.
3. Runtime overhead
CUDA context, kernels, activation and scratch buffers. Budget 0.5-1.5 GB. Ollama and llama.cpp sit at the low end; vLLM sits far higher because it pre-reserves a share of the card for batched serving.
Mixture-of-Experts caveat: an MoE model needs memory for every expert, not just the active ones. Qwen3-Coder 480B activates about 35B parameters per token, but the weights that must be resident are all 480B: 480 x 0.5625 = 270 GB at Q4_K_M. GPT-OSS 120B works out at 120 x 0.5625 = 67.5 GB. Active-parameter counts predict speed, not memory.
How much does each quantization level actually save?
"4-bit" is never exactly 4 bits. Every GGUF block format stores scale factors alongside the weights, so the real cost per weight is a little higher than the name implies. Bytes per weight below is bits divided by 8; the 7B and 70B columns are that figure multiplied by the parameter count.
| Quantization | Bits/weight | Bytes/param | 7B weights | 70B weights | When to pick it |
|---|---|---|---|---|---|
| Q2_K | 2.6 | 0.325 | 2.3 GB | 22.8 GB | Last resort. Only worth it to squeeze a big model onto a small card. |
| Q3_K_M | 3.4 | 0.425 | 3.0 GB | 29.8 GB | Usable on 30B+ models; noticeably rougher on anything under 8B. |
| Q4_K_M | 4.5 | 0.5625 | 3.9 GB | 39.4 GB | The default. llama.cpp marks it "recommended" in its own quantize tool. |
| Q5_K_M | 5.5 | 0.6875 | 4.8 GB | 48.1 GB | About 22% more memory than Q4_K_M for a modest quality gain. |
| Q6_K | 6.6 | 0.825 | 5.8 GB | 57.8 GB | Rarely the right trade — Q8_0 is close in size and closer to FP16. |
| Q8_0 | 8.5 | 1.0625 | 7.4 GB | 74.4 GB | Effectively lossless. Choose it when memory is not the constraint. |
| FP16 | 16 | 2.0 | 14.0 GB | 140.0 GB | The baseline every quantization is measured against. vLLM default. |
Where the odd numbers come from: Q8_0 stores an 8-bit weight plus one FP16 scale per 32-weight block, so 8 + 16/32 = 8.5 bits per weight. Q4_0 is 4 + 16/32 = 4.5. The K-quants add a second-level scale and mix higher-precision tensors into the _M variants, which is why a real GGUF file often lands 5-10% above the table. Format details are documented in the llama.cpp k-quants pull request and the Hugging Face GGUF specification. For a side-by-side of the competing formats, see AWQ vs GPTQ vs GGUF, or run the quantization calculator to compare two levels directly.
What size model fits on my GPU?
Work backwards from the card. Subtract about 1 GB of runtime overhead and a 4K-context KV cache from your VRAM, then divide the remainder by 0.5625 bytes per weight. That gives the largest dense model you can hold at Q4_K_M.
| VRAM | Typical cards | Max dense @ Q4_K_M | What that means in practice |
|---|---|---|---|
| 6 GB | RTX 4050 laptop, RTX 2060 6GB | ~8B | Llama 3.1 8B at Q4_K_M lands at ~6.0 GB total — right at the edge. A 7B is the comfortable pick. |
| 8 GB | RTX 4060, RTX 3070 | ~11B | 8B models fit with room for an 8K context. 14B needs Q3 and gets tight. |
| 12 GB | RTX 3060 12GB, RTX 5070 | ~18B | The cheapest tier that runs a 14B comfortably at Q4_K_M with an 8K context. |
| 16 GB | RTX 4080, RTX 5080, M4 Pro | ~25B | Either a 14B at Q8_0 or a 24B at Q4_K_M — not both. |
| 24 GB | RTX 3090, RTX 4090 | ~39B | The 32B tier. A 32B at Q4_K_M totals ~20 GB, leaving headroom for 16K context. |
| 32 GB | RTX 5090 | ~52B | Runs a 32B at Q5_K_M or Q6_K instead of Q4, or a 32B at Q4 with a very long context. |
| 48 GB | 2x RTX 3090 or 2x RTX 4090 | ~79B | The entry point for a 70B at Q4_K_M (~41.7 GB total). Needs a board with two usable x8 slots. |
| 64 GB unified | Mac Studio M4 Max | ~70B | macOS does not hand the whole pool to the GPU by default — budget roughly 70% unless you raise iogpu.wired_limit_mb. |
| 96-192 GB unified | Mac Studio M3 Ultra | 70B at Q8_0 | The only single-box consumer option for a 70B at near-lossless quantization, or a 120B-class MoE at Q4. |
Each VRAM figure links to the ranked model list for that tier. Splitting across two cards is covered in two GPUs vs one for local LLMs; Apple unified-memory budgeting has its own Apple Silicon memory calculator. Capacity is only half the story — memory bandwidth is what sets tokens per second once the model fits.
How much VRAM does each popular model need?
Weight-only footprints, computed as parameters multiplied by bytes per weight. The Q4 total column adds a 4K-context KV cache and 1 GB of overhead — that is the number to compare against your card.
| Model | Params | Q4_K_M | Q8_0 | FP16 | Q4 total @ 4K | Smallest GPU that fits |
|---|---|---|---|---|---|---|
| Llama 3.2 3B | 3B | 1.7 GB | 3.2 GB | 6.0 GB | ~3.2 GB | Any 4 GB GPU |
| Phi-3.5 Mini | 3.8B | 2.1 GB | 4.0 GB | 7.6 GB | ~3.6 GB | Any 6 GB GPU |
| Mistral 7B | 7B | 3.9 GB | 7.4 GB | 14.0 GB | ~5.5 GB | RTX 3050 8GB |
| Llama 3.1 8B | 8B | 4.5 GB | 8.5 GB | 16.0 GB | ~6.0 GB | RTX 4060 8GB |
| Qwen 2.5 14B | 14B | 7.9 GB | 14.9 GB | 28.0 GB | ~9.7 GB | RTX 3060 12GB |
| Qwen 2.5 32B | 32B | 18.0 GB | 34.0 GB | 64.0 GB | ~20.1 GB | RTX 3090 24GB |
| Llama 3.3 70B | 70B | 39.4 GB | 74.4 GB | 140.0 GB | ~41.7 GB | 2x RTX 3090 |
| Llama 4 Scout | 109B MoE | 61.3 GB | 115.8 GB | 218.0 GB | ~63.1 GB | H100 80GB / 96GB Mac |
Parameter counts are rounded to the model family name (Llama 3.1 8B is 8.03B, Mistral 7B is 7.24B), so real GGUF files land within a few percent of these figures. The Q4 total adds that model's own 4K KV cache from the table below plus 1 GB of overhead — for Llama 3.1 8B: 4.5 + 0.54 + 1 = 6.0 GB. Llama 4 Scout is MoE, so its Q4 column covers all 109B parameters, not the 17B active per token. For the exact download size of a given Ollama tag, use the Ollama model RAM and VRAM table, and see current VRAM requirements by model class for the wider picture. Choosing hardware from scratch? Start with the RTX 5090 vs 5080 comparison.
How much VRAM does context length add?
The KV cache holds one key and one value vector per layer per token. Its size per token is2 x layers x KV heads x head dimension x 2 bytesat FP16 — note that parameter count never appears in that formula. A 70B model with 80 layers costs 2.5x more per token than an 8B model with 32 layers, even though it has nearly nine times the parameters. Every row below is computed from the model's published configuration.
| Model | Attention shape | Per token | 4K context | 128K context |
|---|---|---|---|---|
| Llama 3.2 3B | 28 layers, 8 KV heads, head dim 128 | 112 KiB | 0.44 GiB | 14.0 GiB |
| Llama 3.1 8B | 32 layers, 8 KV heads, head dim 128 | 128 KiB | 0.50 GiB | 16.0 GiB |
| Qwen 2.5 14B | 48 layers, 8 KV heads, head dim 128 | 192 KiB | 0.75 GiB | 24.0 GiB |
| Llama 3.3 70B | 80 layers, 8 KV heads, head dim 128 | 320 KiB | 1.25 GiB | 40.0 GiB |
Worked example for Llama 3.1 8B: 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB per token. At 4,096 tokens that is 0.5 GiB; at 131,072 tokens it is 16 GiB — more than the weights themselves at Q4_K_M. Grouped-query attention is what keeps these numbers survivable: without it, 32 KV heads instead of 8 would make every figure four times larger. Both llama.cpp and vLLM can store the cache at 8 bits instead of 16, which halves the column.
Common VRAM questions, answered
How much VRAM do I need for a 7B parameter model?
About 5.5 GB at Q4_K_M. The arithmetic: 7 billion parameters x 0.5625 bytes per parameter = 3.9 GB of weights, plus roughly 0.5 GB of KV cache at a 4K context and about 1 GB of runtime overhead. At Q8_0 the weights alone are 7.4 GB, and at FP16 they are 14 GB. That puts a Q4 7B model inside any 8 GB card, and an FP16 7B model out of reach of every consumer GPU below 16 GB.
How is VRAM calculated for an LLM?
VRAM = (parameters x bytes per weight) + KV cache + runtime overhead. Bytes per weight comes from the quantization: FP16 is 2.0, Q8_0 is about 1.06, and Q4_K_M is about 0.56. KV cache is 2 x layers x KV heads x head dimension x context length x 2 bytes at FP16. Runtime overhead — CUDA kernels, activation buffers, the framework itself — is typically 0.5 to 1.5 GB. Mixture-of-Experts models need weight memory for every expert, not only the active ones.
What is the best quantization level for local AI?
Q4_K_M for almost everyone. It is the level llama.cpp itself labels as the recommended balance in its quantize tool, and it cuts weight memory by roughly 72 percent versus FP16 (4.5 bits of 16). Q5_K_M costs about 22 percent more memory for a smaller quality gap, and Q8_0 is close to indistinguishable from FP16 but needs nearly double the memory of Q4. Drop to Q3_K_M or Q2_K only when the model will not otherwise fit — the quality cost grows fastest on models under about 8B.
Can I run a model larger than my VRAM?
Yes, through CPU offloading. Ollama and llama.cpp split the model by layer: layers that fit run on the GPU, the remainder run on the CPU against system RAM. Expect throughput to fall at least in proportion to the share of layers on the CPU, and usually worse, because system DDR bandwidth is roughly an order of magnitude below GDDR6X or HBM. Apple Silicon avoids the split entirely — CPU and GPU share one pool of unified memory.
Does context length affect VRAM usage?
Substantially, and it is the reason models that "fit" still crash. The KV cache grows linearly with context. Llama 3.1 8B stores 2 x 32 layers x 8 KV heads x 128 head dimension x 2 bytes = 128 KiB per token, so 4K context costs 0.5 GiB, 32K costs 4 GiB and 128K costs 16 GiB. Llama 3.3 70B has 80 layers, so the same contexts cost 1.25, 10 and 40 GiB. Move the context slider in the calculator to see it.
Is there a VRAM calculator that works for Ollama models?
This one does. Ollama ships GGUF builds and defaults to Q4_K_M for most tags, which is the default quantization here too, so the number the calculator returns is the number ollama ps will report for a matching tag at the same context. Ollama also defaults to a smaller context window than the model maximum, so if you have raised num_ctx, raise the context slider to match before you trust the total.
Why does my model use more VRAM than the calculator says?
Three usual causes. The GGUF file is larger than the nominal bits-per-weight suggests, because llama.cpp keeps token embeddings and the output tensor at higher precision. Your inference server reserved memory ahead of time — vLLM pre-allocates a fraction of the card by default, and that is a setting, not a leak. Or the context is larger than you think: a serving front end may request the model maximum rather than the default.
Still not sure whether your machine is up to it? Run can I run local AI for a whole-system check, or the model recommender to go straight to a model that fits.
Free to use — just keep the attribution link. Works on any site.
<iframe src="https://localaimaster.com/embed/vram-calculator" width="100%" height="560" style="border:1px solid var(--line);border-radius:12px;max-width:680px" title="VRAM Calculator — Local AI Master" loading="lazy"></iframe>
<p style="font:13px/1.5 system-ui,sans-serif;max-width:680px;margin:6px 0 0"><a href="https://localaimaster.com/tools/vram-calculator">VRAM Calculator</a> by <a href="https://localaimaster.com">Local AI Master</a></p>Know what to actually run on it
All 561 chapters — running local models, RAG, agents, fine-tuning — plus the Python Lab and every course added later.
Ready to Go Beyond Tutorials?
25 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?
Go from reading about AI to building with AI
25 structured courses. Hands-on projects. Runs on your machine. Start free.
Written by the Local AI Master Team
The team behind Local AI Master
We build Local AI Master around practical, testable local AI workflows: model selection, hardware planning, RAG systems, agents, and MLOps. The goal is to turn scattered tutorials into a structured learning path you can follow on your own hardware.
Related Guides
Continue your local AI journey with these comprehensive guides
Grab the AI Starter Kit — career roadmap, cheat sheet, setup guide
No spam. Unsubscribe with one click.