Best LLM for 32GB VRAM (32GB VRAM)
RTX 5090 territory — full 32Bs at speed, and a 70B that (barely) fits
The best local LLM for 32GB VRAM in 2026 is Qwen3.6 27B — ~17GB at Q4_K_M and 60–90 tokens/sec on an RTX 5090, with headroom 24GB owners can only envy. The tier’s real unlock is comfort: every dense 32B runs at ~45 tok/s with a long context, and Llama 3.3 70B squeezes in at Q2_K (~26GB) — the first time a single consumer card holds a 70B at all. For daily work, a 27–32B at Q4 still beats a 70B at Q2.
Models that fit in 32GB VRAM
Tested reference: RTX 5090 (32GB). Figures are for Q4_K_M with a modest context window.
| Model | Size | Build | VRAM used | Speed |
|---|---|---|---|---|
| WINNERQwen3.6 27B Best overall — top-tier agentic coding (68.9% SWE-bench Verified) at interactive speed, with ~15GB left for context or a second model. ollama pull qwen3.6:27b | 27B (dense) | Q4_K_M | ~17GB | ~60–90 tok/s |
| Qwen 2.5 32B The 32B generalist finally gets a full-length context — no more rationing KV cache like on 24GB. ollama pull qwen2.5:32b | 32B | Q4_K_M | ~20GB | ~45 tok/s |
| DeepSeek-R1 32B Chain-of-thought reasoning at full 32B quality — thinking tokens are cheap at this speed. ollama pull deepseek-r1:32b | 32B | Q4_K_M | ~20GB | ~45 tok/s |
| Qwen 2.5 Coder 32B The SOTA dense local coder, now with room for a real repo context beside it. ollama pull qwen2.5-coder:32b | 32B | Q4_K_M | ~20GB | ~45 tok/s |
| Llama 3.3 70B (Q2_K) The capability flex: a 70B fully in VRAM on one consumer card. Q2_K costs real quality — benchmark it against the 32Bs on your own tasks before adopting it. ollama pull llama3.3:70b-instruct-q2_K | 70B | Q2_K | ~26GB | ~25–30 tok/s (est.) |
What won't fit in 32GB
- ✗Llama 3.3 70B (Q4_K_M) (needs ~42GB) — Full-quality 70B still needs 48GB — two 24GB cards or two 5090s.
- ✗Qwen3-Coder-Next 80B (needs ~52GB (Q4)) — The big MoE coder is H100 or dual-5090 territory.
- ✗gpt-oss:120b (needs ~65GB+) — Workstation class — an 80GB card or a 64GB+ Mac with offload.
How to fit more in 32GB
- →32GB’s best trick is not the 70B — it is pairing models: Qwen3.6 27B (~17GB) and Flux.1 dev FP8 (~11GB) fit together, so an agent can write and render images without swapping models.
- →The step up from 24GB buys context more than parameters: a ~20GB 32B leaves ~12GB for KV cache here versus ~4GB on a 4090 — with flash attention that is a 32K+ context without spilling.
- →If you do run the 70B, keep expectations honest: Q2_K visibly degrades output, and Q3_K_S (~31GB) leaves no room for KV cache on a 32GB card. A 32B at Q4 wins most head-to-heads — test before you commit.
Quick start
curl -fsSL https://ollama.com/install.sh | shollama run qwen3.6:27bGo from "it runs" to actually building
All 519 chapters — running local models, RAG, agents, fine-tuning — plus the Python Lab and every course added later.
Frequently asked questions
Can an RTX 5090 run a 70B model?
Just barely. Llama 3.3 70B at Q2_K is ~26GB, which fits with a modest context — the first single consumer card to hold a 70B entirely in VRAM. But Q2 quantization costs visible quality, and the Q3_K_S build (~31GB) leaves no context room. For serious 70B work you still want 48GB; on one 5090 a dense 32B at Q4 usually answers better.
Is the RTX 5090 worth it over a 24GB card for local AI?
If you live in the 27–32B class, yes: the same models run measurably faster (~45 tok/s for a 32B at Q4 in Ollama, vs ~28 on an RTX 4090) and the extra 8GB converts into a genuinely long context instead of a rationed one. If your daily models are 14B or smaller, a 16–24GB card already runs them perfectly — the 5090 buys speed, not capability.
What speed does a 32B model reach on an RTX 5090?
Around 45 tok/s at Q4_K_M in Ollama — published single-5090 benchmarks measured Qwen 2.5 32B at 45.1 and DeepSeek-R1 32B at 45.5. The dense Qwen3.6 27B lands at 60–90 tok/s depending on engine and context length.
Related guides
Ready to Go Beyond Tutorials?
20 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?