AI VRAM Requirements: How Much GPU Memory for Every Model

VRAM Requirements by Model Size

Quick Reference Table

VRAM Formula

VRAM (GB) = Parameters (B) × Bytes_per_param × 1.2

Bytes per param:
- FP16/BF16: 2 bytes
- Q8_0: 1 byte
- Q5_K_M: 0.7 bytes
- Q4_K_M: 0.55 bytes

Quantization Impact

What You Lose at Each Level

Recommendation: Q4_K_M is the sweet spot—significant savings with minimal quality loss.

Context Window VRAM

Context length adds to base VRAM requirements:

Formula: ~(context² × layers × 2) / 1e9 GB

GPU Recommendations by Use Case

Casual Use / Learning

RTX 4060 8GB ($299)

• Runs: 7B models comfortably
• Use: Learning, simple chat

Hobbyist

RTX 4070 Ti Super 16GB ($799)

• Runs: 14B-32B models, Mixtral
• Use: Daily AI assistant, coding help

Power User

RTX 4090 24GB ($1,599)

• Runs: 70B Q4, most models
• Use: Serious local AI, development

Professional

RTX 5090 32GB ($1,999)

• Runs: 70B Q5/Q8, larger contexts
• Use: Production, enterprise

Enterprise

Dual RTX 4090 48GB ($3,200)

• Runs: 70B Q8, 120B+ models
• Use: Large models, training

What Fits on Your GPU?

8GB VRAM (RTX 4060, 4070)

16GB VRAM (RTX 4070 Ti Super, 4080)

24GB VRAM (RTX 4090, 5090)

Optimizing VRAM Usage

1. Choose Right Quantization

# Q4 for 70B on 24GB
ollama run llama3.1:70b-q4_K_M

# Q5 if you have headroom
ollama run llama3.1:70b-q5_K_M

2. Reduce Context

# Default context uses more VRAM
ollama run model

# Reduced context saves VRAM
ollama run model --num-ctx 4096

3. Unload Unused Models

# Keep only active model loaded
ollama stop model_name

4. GPU Layers for Hybrid

# Partial GPU, rest on CPU
OLLAMA_NUM_GPU=30 ollama run model

Multi-GPU Setups

Combining VRAM

Configuration

# Automatic multi-GPU in llama.cpp
./main -m model.gguf -ngl 99 # Uses all GPUs

# Ollama multi-GPU
CUDA_VISIBLE_DEVICES=0,1 ollama serve

Key Takeaways

1. Q4_K_M is the sweet spot for most users
2. 24GB handles most models including 70B
3. Context length adds significant VRAM
4. Multi-GPU helps but with overhead
5. Budget more VRAM than minimum for headroom

Next Steps

1. Browse the best Ollama models — VRAM requirements for every model
2. AWQ vs GPTQ vs GGUF — quantization formats that determine VRAM usage
3. Choose your GPU based on VRAM needs
4. Find models for 8GB RAM — budget hardware recommendations
5. Set up Open WebUI once your hardware is ready

---

VRAM is the key constraint for local AI. Understanding these requirements helps you choose the right hardware and optimize your setup.