Best LLM for 24GB VRAM (24GB VRAM)
What an RTX 4090/3090 really runs — full 32B at Q4
The best local LLM for 24GB VRAM in 2026 is Qwen 2.5 32B — ~20GB at Q4_K_M and ~28 tokens/sec on an RTX 4090, with room for a long context. 24GB is the tier where full 32B models run at Q4 quality: Gemma 2 27B is the best writer, Command-R 35B the best for RAG. A 70B model still needs ~40GB, so 32B is the 24GB ceiling — the sweet spot most serious local-AI builds target.
Models that fit in 24GB VRAM
Tested reference: RTX 4090 / RTX 3090 (24GB). Figures are for Q4_K_M with a modest context window.
| Model | Size | Build | VRAM used | Speed |
|---|---|---|---|---|
| WINNERQwen 2.5 32B Best overall at 24GB — top reasoning/coding in a size that fits one card at full Q4. ollama pull qwen2.5:32b | 32B | Q4_K_M | ~20GB | ~28 tok/s |
| Gemma 2 27B Best writing/prose quality; leaves headroom for a big context. ollama pull gemma2:27b | 27B | Q4_K_M | ~17GB | ~30 tok/s |
| Command-R 35B Built for RAG + tool use with a 128K context and citations. ollama pull command-r | 35B | Q4_K_M | ~21GB | ~25 tok/s |
| Yi 1.5 34B Strong bilingual (EN/ZH) and long-form reasoning. ollama pull yi:34b | 34B | Q4_K_M | ~21GB | ~26 tok/s |
| Qwen 3 14B (Q8) 24GB lets you run a 14B at Q8 (near-FP16 fidelity) with a huge context — great when quality > size. ollama pull qwen3:14b-q8_0 | 14B | Q8_0 | ~16GB | ~32 tok/s |
What won't fit in 24GB
- ✗Llama 3.3 70B (needs ~40GB) — Needs 48GB (or 2×24GB). On a single 24GB card a 70B only runs at Q2 with heavy offload — not worth it vs a 32B at Q4.
- ✗Qwen 2.5 72B (needs ~42GB) — Needs 48GB.
How to fit more in 24GB
- →24GB is the single-card sweet spot: full 32B at Q4, or a 14B at Q8 for near-lossless quality.
- →For 70B, add a second 24GB card (2×3090 is the classic budget 48GB build) rather than crushing it onto one.
- →Flash-attention lets a 32B hold a 32K+ context comfortably in 24GB.
Quick start
curl -fsSL https://ollama.com/install.sh | shollama run qwen2.5:32bGo from “it runs” to actually building
Learn to run local models, wire up RAG, and build agents — hands-on, with a free first chapter of every course.
Start learning free →Frequently asked questions
Can 24GB VRAM run a 70B model?
Not at usable quality — a 70B needs ~40GB at Q4. On one 24GB card it only runs at Q2 with system-RAM offload (a few tok/s). For 70B, use two 24GB cards (48GB). On a single 24GB card, a 32B at Q4 is the right ceiling.
Is 24GB VRAM enough for local AI?
It is the enthusiast sweet spot. 24GB (RTX 4090/3090) runs any 32B model at full Q4 (~28 tok/s), or a 14B at Q8 near-lossless — covering serious coding, RAG and agent work locally.
RTX 4090 vs 3090 for LLMs?
Both have 24GB, so both run the same models. The 4090 is ~1.5–2× faster tok/s; the 3090 is far cheaper used and is the classic value pick (and 2×3090 = a 48GB 70B build).
Related guides
Ready to Go Beyond Tutorials?
20 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?