AI VRAM Requirements 2026: GPU Sizes for 7B, 13B, 70B Models

How much VRAM do you need for AI models? With Q4 quantization, a 7B model needs 4-6GB, a 13B model needs 8-10GB, a 32B model needs ~20GB, and a 70B model needs 40GB+. As a rule of thumb: VRAM (GB) ≈ parameters (in billions) × bytes-per-param (0.5 for Q4, 1 for Q8, 2 for FP16) × 1.2 for overhead. An 8GB GPU comfortably runs 7B-8B models; 16GB handles 14B models and 20B-class MoE models such as gpt-oss 20B; 24GB runs the 27B-32B class at Q4; a 70B at Q4 (a 43GB file) needs 48GB, such as two 24GB cards.

VRAM Requirements by Model Size

Quick Reference Table

VRAM Formula

VRAM (GB) = Parameters (B) × Bytes_per_param × 1.2

Bytes per param:
- FP16/BF16: 2 bytes
- Q8_0: 1 byte
- Q5_K_M: 0.7 bytes
- Q4_K_M: 0.55 bytes

Quantization Impact

What You Lose at Each Level

Savings follow from bits per weight. Quality loss is described rather than scored because it varies by model and task; llama.cpp measures it as the rise in perplexity, and it climbs steeply below 4 bits.

Recommendation: Q4_K_M is the sweet spot—significant savings with minimal quality loss.

Context Window VRAM

Context length adds to base VRAM requirements:

Formula: KV cache ≈ 2 × layers × KV heads × head size × context × 2 bytes (FP16). For Llama 3.1 70B (80 layers, 8 KV heads, head size 128) that is about 0.33MB per token, so the cache grows linearly with context, not quadratically.

GPU Recommendations by Use Case

Casual Use / Learning

RTX 4060 8GB ($299 launch MSRP; check the current price)

• Runs: 7B models comfortably
• Use: Learning, simple chat

Hobbyist

RTX 4070 Ti Super 16GB ($799 launch MSRP; check the current price)

• Runs: 14B models and gpt-oss 20B; 32B models and Mixtral only with CPU offload
• Use: Daily AI assistant, coding help

Power User

RTX 4090 24GB ($1,599 launch MSRP; check the current price)

• Runs: 27B-32B at Q4/Q5; a 70B only with most of it offloaded to system RAM
• Use: Serious local AI, development

Professional

RTX 5090 32GB ($1,999 MSRP; on November 2, 2026 the Founders Edition was out of stock at the NVIDIA marketplace and a GIGABYTE AORUS card was listed at $3,340 at Walmart)

• Runs: 27B-32B at up to Q6, or at Q4 with long contexts (a 32B at Q8 is about 35GB and does not fit); a 70B at Q4 (43GB) still spills about 11GB to system RAM
• Use: Production, enterprise
• Note: memory shortages have kept the 5090 well above its $1,999 MSRP; see GPU prices and the memory shortage.

Sweet Spot for Mid-Range (not released yet)

RTX 5070 Ti SUPER / 5080 SUPER 24GB (leaked, not announced by NVIDIA as of November 2, 2026; no price)

• Would run: 32B at Q4/Q5 and gpt-oss 20B, the same class as any 24GB card
• Status: reporting points to a delay over 3GB GDDR7 memory pricing; until NVIDIA announces the cards, plan around a used RTX 3090 or an RTX 5090 for 24GB+. The latest reporting is in GPU prices and the memory shortage.

Enterprise

Dual RTX 4090 48GB (two cards at $1,599 launch MSRP each; check current prices)

• Runs: 70B at Q4_K_M (43GB file); 70B at Q8 (75GB) and gpt-oss 120B (~65GB) do not fit
• Use: Large models, training

48GB is also the classic QLoRA fine-tuning tier: the QLoRA paper (Dettmers et al., 2023) fine-tuned a 65B model on a single 48GB GPU, and our LoRA fine-tuning guide covers the same method on smaller cards.

Not sure which card matches your budget and target model size? Our interactive which-GPU-to-buy tool maps your VRAM needs to a specific recommendation, and the full best-GPUs-for-AI guide breaks down bandwidth, price, and power for every current card.

What Fits on Your GPU?

8GB VRAM (RTX 4060, 4070)

16GB VRAM (RTX 4070 Ti Super, 4080)

24GB VRAM (RTX 3090, 4090)

File sizes are the Ollama downloads (phi3:14b, mixtral:8x7b, deepseek-r1:32b, gpt-oss:20b, llama3.1:70b-instruct-q4_K_M, deepseek-v3:671b, llama4:maverick, qwen2.5:72b, qwen3:32b, gemma3:27b), read from the Ollama registry on November 2, 2026. Leave 1-2GB on top for the KV cache.

How Much VRAM for 2026's Most Popular Models?

The generic 7B/13B/70B buckets are a great starting point, but the models people actually download in 2026 don't fall neatly on those round numbers — and several are Mixture-of-Experts (MoE), which changes the math. Here are real, current models with approximate VRAM at the quant most people run (Q4_K_M unless noted). Figures include a typical 4K-8K context; budget more for long context (see the KV cache section below).

A few things worth calling out, because they trip people up:

• MoE memory is the total, not the active count. gpt-oss 120B only activates ~5B parameters per token, so it's fast like a small model — but you still have to hold all ~117B weights in memory. The clean single-GPU answer is ~60-65GB (so an 80GB H100, or a 96GB+ workstation card); a 24GB consumer card cannot host it GPU-resident.
• The biggest MoE releases are out of reach for one card. MiniMax M3 (~428B total, ~23B active) is a 257-265GB download even at 4-bit, and Ollama only offers it as a cloud tag. Can you run MiniMax M3 locally? lists every build's size and what can hold it.
• gpt-oss 20B is the new "16GB sweet spot." It runs comfortably on a 16GB card such as the RTX 4080, on an RTX 4090, or on a 24GB Mac and is the most capable open model that fits a single mainstream card in 2026.
• The 32B class (Qwen3 32B, Gemma 3 27B, DeepSeek R1 distills) is the real home for 24GB cards — better quality than 8B, and you keep headroom for context. If your main use is coding, our guide to picking the right model size for coding (7B vs 14B vs 32B vs 70B) walks through where the quality jumps actually matter.

How Does Context Length (KV Cache) Eat Your VRAM?

The model weights are only half the story. Every token you keep in context lives in the KV cache, and on long contexts the cache can rival or exceed the model itself. A rough, current rule: for a 7B-8B model, each ~1,000 tokens of context adds roughly ~0.1GB to VRAM in FP16. Scale that up by model size and context, and a Llama-3-8B at 32K context burns roughly 4GB on KV cache alone — nearly as much as the 4-5GB Q4 weights.

Approximate KV-cache formula (FP16):

KV cache bytes ≈ 2 × layers × kv_heads × head_dim × tokens × 2 bytes

Two modern features dramatically reduce this:

• GQA (Grouped-Query Attention) — used by almost every 2026 model (Llama 3.x, Qwen3, Gemma 3) — cuts KV cache 50-75% by sharing keys/values across query heads, with no quality loss. It's baked into the model, so you get it for free.
• KV cache quantization (FP8 or INT4) — supported in llama.cpp, vLLM, and others — cuts cache memory another 50-75%. This is how people fit 32K+ context into an 8GB card. In llama.cpp, enable it with --cache-type-k q8_0 --cache-type-v q8_0; in Ollama, set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 before starting the server.

If you want exact numbers for a specific model + context + quant combination, plug it into our VRAM calculator tool rather than estimating — it accounts for KV cache, GQA, and overhead so you don't guess wrong on a $1,000+ GPU.

VRAM vs System RAM: When Does RAM Step In?

VRAM is where you want the whole model to live — a 360-1,008 GB/s graphics card moves weights roughly 4-11× faster than dual-channel DDR5-5600 (89.6 GB/s), and generation speed follows bandwidth. But when a model doesn't fit, Ollama and llama.cpp spill the overflow layers to system RAM (CPU offload), which works but is much slower for the offloaded portion. That makes system RAM your safety net, especially for MoE giants and partial-offload setups. If you're sizing a build, read our companion RAM requirements for local AI guide alongside this one — the two budgets are different, and people routinely over-buy VRAM while starving the system RAM that long contexts and CPU-offload actually need.

Optimizing VRAM Usage

1. Choose Right Quantization

# Q4 for 70B: a 43GB file, so 48GB of VRAM (on 24GB, most layers spill to system RAM)
ollama run llama3.1:70b-instruct-q4_K_M

# Q5 if you have headroom: a 50GB file
ollama run llama3.1:70b-instruct-q5_K_M

2. Reduce Context

# Default context uses more VRAM
ollama run model

# Reduced context saves VRAM: set it inside the session
ollama run model
>>> /set parameter num_ctx 4096
# or for every model the server loads
OLLAMA_CONTEXT_LENGTH=4096 ollama serve

3. Unload Unused Models

# Keep only active model loaded
ollama stop model_name

4. GPU Layers for Hybrid

# Partial GPU, rest on CPU: num_gpu is the number of layers on the GPU
ollama run model
>>> /set parameter num_gpu 30

Multi-GPU Setups

Combining VRAM

Configuration

# Automatic multi-GPU in llama.cpp
llama-cli -m model.gguf -ngl 99 # Uses all GPUs

# Ollama multi-GPU
CUDA_VISIBLE_DEVICES=0,1 ollama serve

Key Takeaways

1. Q4_K_M is the sweet spot for most users
2. 24GB handles the 27B-32B class; a 70B at Q4 needs 48GB
3. Context length adds significant VRAM
4. Multi-GPU helps but with overhead
5. Budget more VRAM than minimum for headroom

Next Steps

1. Browse the best Ollama models — VRAM requirements for every model
2. AWQ vs GPTQ vs GGUF — quantization formats that determine VRAM usage
3. Choose your GPU based on VRAM needs
4. Find models for 8GB RAM — budget hardware recommendations
5. Set up Open WebUI once your hardware is ready

---

VRAM is the key constraint for local AI. Understanding these requirements helps you choose the right hardware and optimize your setup.