Best LLM for 32GB VRAM (32GB VRAM)
RTX 5090 territory — full 32Bs at speed, and a 70B that (barely) fits
The best local LLM for 32GB VRAM in 2026 is Qwen3.6 27B — ~17GB at Q4_K_M, with a bandwidth ceiling of ~105 tokens/sec on an RTX 5090 (1,792 GB/s ÷ 17GB — a ceiling, not a measurement) and headroom 24GB owners can only envy. The tier’s real unlock is comfort: every dense 32B fits with a long context and a ceiling near 90 tok/s, and Llama 3.3 70B squeezes in at Q2_K (~26GB) — the first time a single consumer card holds a 70B at all. For daily work, a 27–32B at Q4 still beats a 70B at Q2.
Models that fit in 32GB VRAM
Reference card: RTX 5090 (32GB). Figures are for Q4_K_M with a modest context window.
| Model | Size | Build | VRAM used | Ceiling tok/s* |
|---|---|---|---|---|
| WINNERQwen3.6 27B Best overall — top-tier agentic coding (77.2% SWE-bench Verified on Qwen’s own model card) at interactive speed, with ~15GB left for context or a second model. ollama pull qwen3.6:27b | 27B (dense) | Q4_K_M | ~17GB | ceiling ~105 tok/s (1,792 ÷ 17GB) |
| Qwen 2.5 32B The 32B generalist finally gets a full-length context — no more rationing KV cache like on 24GB. ollama pull qwen2.5:32b | 32B | Q4_K_M | ~20GB | ceiling ~90 tok/s (1,792 ÷ 20GB) |
| DeepSeek-R1 32B Chain-of-thought reasoning at full 32B quality — thinking tokens are cheap at this speed. ollama pull deepseek-r1:32b | 32B | Q4_K_M | ~20GB | ceiling ~90 tok/s (1,792 ÷ 20GB) |
| Qwen 2.5 Coder 32B The SOTA dense local coder, now with room for a real repo context beside it. ollama pull qwen2.5-coder:32b | 32B | Q4_K_M | ~20GB | ceiling ~90 tok/s (1,792 ÷ 20GB) |
| Llama 3.3 70B (Q2_K) The capability flex: a 70B fully in VRAM on one consumer card. Q2_K costs real quality — benchmark it against the 32Bs on your own tasks before adopting it. ollama pull llama3.3:70b-instruct-q2_K | 70B | Q2_K | ~26GB | ceiling ~69 tok/s (1,792 ÷ 26GB) |
* Ceiling tok/s = the card’s vendor-listed memory bandwidth ÷ the model’s file size as Ollama lists it — the rate a dense model cannot exceed, because every generated token reads the whole weight set once. We have not measured these cards. Real-world generation usually lands at 50–80% of the ceiling and falls as the context grows; mixture-of-experts rows read only their active parameters, so no ceiling is quoted for them.
What won't fit in 32GB
- ✗Llama 3.3 70B (Q4_K_M) (needs ~42GB) — Full-quality 70B still needs 48GB — two 24GB cards or two 5090s.
- ✗Qwen3-Coder-Next 80B (needs ~52GB (Q4)) — The big MoE coder is H100 or dual-5090 territory.
- ✗gpt-oss:120b (needs ~65GB+) — Workstation class — an 80GB card or a 64GB+ Mac with offload.
How to fit more in 32GB
- →32GB’s best trick is not the 70B — it is pairing models: Qwen3.6 27B (~17GB) and Flux.1 dev FP8 (~11GB) fit together, so an agent can write and render images without swapping models.
- →The step up from 24GB buys context more than parameters: a ~20GB 32B leaves ~12GB for KV cache here versus ~4GB on a 4090 — with flash attention that is a 32K+ context without spilling.
- →If you do run the 70B, keep expectations honest: Q2_K visibly degrades output, and Q3_K_S (~31GB) leaves no room for KV cache on a 32GB card. A 32B at Q4 wins most head-to-heads — test before you commit.
Quick start
curl -fsSL https://ollama.com/install.sh | shollama run qwen3.6:27bGo from "it runs" to actually building
All 561 chapters — running local models, RAG, agents, fine-tuning — plus the Python Lab and every course added later.
Frequently asked questions
Can an RTX 5090 run a 70B model?
Just barely. Llama 3.3 70B at Q2_K is ~26GB, which fits with a modest context — the first single consumer card to hold a 70B entirely in VRAM. But Q2 quantization costs visible quality, and the Q3_K_S build (~31GB) leaves no context room. For serious 70B work you still want 48GB; on one 5090 a dense 32B at Q4 usually answers better.
Is the RTX 5090 worth it over a 24GB card for local AI?
If you live in the 27–32B class, yes: the same models get a far higher bandwidth ceiling (1,792 vs 1,008 GB/s — ~90 vs ~50 tok/s on a 20GB 32B) and the extra 8GB converts into a genuinely long context instead of a rationed one. If your daily models are 14B or smaller, a 16–24GB card already runs them perfectly — the 5090 buys speed, not capability.
What speed does a 32B model reach on an RTX 5090?
We have not measured one, so we will not quote a rate as if we had. The ceiling is 1,792 GB/s ÷ 20GB ≈ 90 tok/s for a 32B at Q4_K_M, and ≈ 105 for the 17GB Qwen3.6 27B; real engines land below that and lower still as the context grows. Any page quoting an exact 5090 figure without naming the rig, engine, quant and context length is guessing.
Related guides
Ready to Go Beyond Tutorials?
25 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?