Best LLM for 24GB VRAM (24GB VRAM)
What an RTX 4090/3090 really runs — full 32B at Q4
The best local LLM for 24GB VRAM in 2026 is Qwen 2.5 32B — ~20GB at Q4_K_M, with a bandwidth ceiling of ~50 tokens/sec on an RTX 4090 (1,008 GB/s ÷ 20GB — a ceiling, not a measurement) and room for a long context. 24GB is the tier where full 32B models run at Q4 quality: Gemma 2 27B is the best writer, Command-R 35B the best for RAG. A 70B model still needs ~40GB, so 32B is the 24GB ceiling — the sweet spot most serious local-AI builds target.
Models that fit in 24GB VRAM
Reference card: RTX 4090 / RTX 3090 (24GB). Figures are for Q4_K_M with a modest context window.
| Model | Size | Build | VRAM used | Ceiling tok/s* |
|---|---|---|---|---|
| WINNERQwen 2.5 32B Best overall at 24GB — top reasoning/coding in a size that fits one card at full Q4. ollama pull qwen2.5:32b | 32B | Q4_K_M | ~20GB | ceiling ~50 tok/s (1,008 ÷ 20GB) |
| Gemma 2 27B Best writing/prose quality; leaves headroom for a big context. ollama pull gemma2:27b | 27B | Q4_K_M | ~16GB | ceiling ~63 tok/s (1,008 ÷ 16GB) |
| Command-R 35B Built for RAG + tool use with a 128K context and citations. ollama pull command-r | 35B | Q4_K_M | ~19GB | ceiling ~53 tok/s (1,008 ÷ 19GB) |
| Yi 1.5 34B Strong bilingual (EN/ZH) and long-form reasoning. ollama pull yi:34b | 34B | Q4_K_M | ~19GB | ceiling ~53 tok/s (1,008 ÷ 19GB) |
| Qwen 3 14B (Q8) 24GB lets you run a 14B at Q8 (near-FP16 fidelity) with a huge context — great when quality > size. ollama pull qwen3:14b-q8_0 | 14B | Q8_0 | ~16GB | ceiling ~63 tok/s (1,008 ÷ 16GB) |
* Ceiling tok/s = the card’s vendor-listed memory bandwidth ÷ the model’s file size as Ollama lists it — the rate a dense model cannot exceed, because every generated token reads the whole weight set once. We have not measured these cards. Real-world generation usually lands at 50–80% of the ceiling and falls as the context grows; mixture-of-experts rows read only their active parameters, so no ceiling is quoted for them.
What won't fit in 24GB
- ✗Llama 3.3 70B (needs ~40GB) — Needs 48GB (or 2×24GB). On a single 24GB card a 70B only runs at Q2 with heavy offload — not worth it vs a 32B at Q4.
- ✗Qwen 2.5 72B (needs ~42GB) — Needs 48GB.
How to fit more in 24GB
- →24GB is the single-card sweet spot: full 32B at Q4, or a 14B at Q8 for near-lossless quality.
- →For 70B, add a second 24GB card (2×3090 is the classic budget 48GB build) rather than crushing it onto one.
- →Flash-attention lets a 32B hold a 32K+ context comfortably in 24GB.
Quick start
curl -fsSL https://ollama.com/install.sh | shollama run qwen2.5:32bGo from "it runs" to actually building
All 561 chapters — running local models, RAG, agents, fine-tuning — plus the Python Lab and every course added later.
Frequently asked questions
Can 24GB VRAM run a 70B model?
Not at usable quality — a 70B needs ~40GB at Q4. On one 24GB card it only runs at Q2 with system-RAM offload (system-RAM pace). For 70B, use two 24GB cards (48GB). On a single 24GB card, a 32B at Q4 is the right ceiling.
Is 24GB VRAM enough for local AI?
It is the enthusiast sweet spot. 24GB (RTX 4090/3090) runs any 32B model at full Q4 (bandwidth ceiling ~50 tok/s on a 4090), or a 14B at Q8 near-lossless — covering serious coding, RAG and agent work locally.
RTX 4090 vs 3090 for LLMs?
Both have 24GB, so both run the same models. The 4090 is the faster card, though on token generation their ceilings are only ~8% apart (1,008 vs 936 GB/s) — the bigger gap is prompt processing, which we have not measured; the 3090 is far cheaper used and is the classic value pick (and 2×3090 = a 48GB 70B build).
Related guides
Ready to Go Beyond Tutorials?
25 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?