Best Qwen Model for 12GB VRAM: What Fits an RTX 3060 (12GB VRAM)
The rung between the 8GB and 16GB guides — and the KV-cache maths that decides it
The best Qwen model for 12GB VRAM in 2026 is Qwen3.5-9B — 5.29 GiB at Q4_K_M in the unsloth/Qwen3.5-9B-GGUF listing, small enough that a 12GB card can run it at Q8_0 (8.87 GiB) instead of settling for 4-bit. The reason it wins is not the parameter count, it is the attention layout: Qwen’s model card gives the 9B a hidden layout of 8 × (3 × Gated DeltaNet → 1 × Gated Attention), so only 8 of its 32 layers keep a growing KV cache. Qwen 3 14B at Q4_K_M (8.38 GiB, Qwen’s own GGUF repo) has more parameters but all 40 layers are full attention, which costs 160 KiB per token and eats the rest of the card by ~16K of context. For code specifically, Qwen 2.5 Coder 14B (8.37 GiB) is still the largest Qwen coder that fits. Every 27B Qwen — 3.5, 3.6 and 3.8 — needs 16–24GB.
Models that fit in 12GB VRAM
Tested reference: RTX 3060 12GB / RTX 4070 / RTX 5070. Figures are for Q4_K_M with a modest context window.
| Model | Size | Build | VRAM used | Speed |
|---|---|---|---|---|
| WINNERQwen3.5-9B (Q4_K_M) The pick. 5.68 GB on disk in unsloth/Qwen3.5-9B-GGUF, which is 5.29 GiB. Apache 2.0, 262,144-token native context, image input, and Qwen’s own model card puts it at 82.5 MMLU-Pro and 81.7 GPQA Diamond — above the GPT-OSS-20B column in the same table. Half a 12GB card, so the context budget is the part you get to spend. ollama pull qwen3.5:9b | 9B (dense, vision) | Q4_K_M GGUF | 5.29 GiB | fast · ~5.3 GiB/token |
| Qwen3.5-9B (Q8_0) The 12GB flex nobody at 8GB gets: near-lossless weights of the newest small Qwen, 9.53 GB on disk. At 32K context the KV cache adds 1 GiB (32 KiB/token × 32,768), so 8.87 + 1 ≈ 9.9 GiB — it fits, with the display still to pay for. Not available as an Ollama tag; pull the GGUF. unsloth/Qwen3.5-9B-GGUF · Q8_0 | 9B (dense, vision) | Q8_0 GGUF | 8.87 GiB | slowest here · ~8.9 GiB/token |
| Qwen 3 14B (Q4_K_M) The more-parameters answer, and the one most 12GB lists stop at. 9.00 GB in Qwen’s own Qwen3-14B-GGUF repo. Its 40 layers are all full attention with 8 KV heads at head_dim 128, so context costs 160 KiB/token — roughly 1.25 GiB at 8K and 5 GiB at 32K. On 12GB that is the ceiling, not the weights. ollama pull qwen3:14b | 14B (dense) | Q4_K_M GGUF | 8.38 GiB | slow · ~8.4 GiB/token |
| Qwen 2.5 Coder 14B (Q4_K_M) Still the largest Qwen coder that fits 12GB — Qwen’s 2026 coders start at Qwen3-Coder 30B A3B, which Ollama lists at 19GB. 8.99 GB on disk. Its 48 layers cost 192 KiB/token, the most on this page, so feed it focused files rather than a whole repo. ollama pull qwen2.5-coder:14b | 14B (dense) | Q4_K_M GGUF | 8.37 GiB | slow · ~8.4 GiB/token |
| Qwen3.5-4B (Q8_0) The agent-loop pick: same 32 KiB/token context cost as the 9B, half the bytes per token, and 79.9 on TAU2-Bench in Qwen’s reported table — level with the 9B on that benchmark. At Q8 on a 12GB card you are running it at close to full fidelity and can still afford a 128K window (32 KiB × 131,072 = 4 GiB). ollama pull qwen3.5:4b | 4B (dense, vision) | Q8_0 GGUF | 4.17 GiB | fastest · ~4.2 GiB/token |
What won't fit in 12GB
- ✗Qwen3.8-27B (UD-Q3_K_XL) (needs 12.24 GiB) — The newest Qwen 27B, and the smallest 3-bit build worth running — 13.15 GB on disk, which is 12.24 GiB, already past a 12GB card before a single token of context. The builds that do come in under 12 GiB are 2-bit and low-3-bit (UD-Q2_K_XL 9.15 GiB, UD-IQ3_S 11.21 GiB); they load and leave nothing for KV. This is a 16–24GB model.
- ✗Qwen3.6-27B (needs 17GB) — Ollama lists qwen3.6:27b at 17GB and qwen3.6:35b-a3b at 24GB. Both are 24GB-card models.
- ✗Qwen3.5-27B (needs 17GB) — The dense 27B of the same generation as the 9B above — same architecture, three times the weights, one tier up.
- ✗Qwen3-Coder 30B A3B (needs 19GB) — The MoE coder with the 256K native context. It is the reason people buy 24GB cards for local coding.
- ✗Qwen 2.5 Coder 32B (needs 20GB) — The old SOTA dense coder — still a 24GB pull.
How to fit more in 12GB
- →The "speed" column is bytes read per generated token, i.e. the resident size. Local decode is memory-bandwidth bound, so tok/s ≈ your card’s bandwidth ÷ that number — halve the bytes and you roughly double the rate. No absolute tok/s is quoted on this page because none is published for these models on a 12GB card, and inventing one would be worse than leaving it out.
- →Work out your context budget before you pick a quant. KV cache per token = layers × 2 (K and V) × KV heads × head_dim × 2 bytes at fp16. Qwen 3 14B: 40 × 2 × 8 × 128 × 2 = 160 KiB/token. Qwen3.5-9B: only its 8 gated-attention layers count, 8 × 2 × 4 × 256 × 2 = 32 KiB/token. That 5× gap is the whole argument for the 9B on a card this size.
- →Ollama’s listed size is not the weight size for Qwen3.5. qwen3.5:9b shows as 6.6GB because it bundles the 0.92 GB vision projector on top of the 5.68 GB Q4_K_M weights (5.68 + 0.92 = 6.60). Text-only in llama.cpp, you are carrying 5.29 GiB.
- →Do not set num_ctx to the advertised 262,144. At 32 KiB/token that window alone is 8 GiB of KV cache. Ollama’s own docs default anything under 24 GiB of VRAM to a 4K context for exactly this reason; 32–64K is the realistic ask on 12GB, and OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention halves it again.
- →Prefer a Q8 9B to a 3-bit 27B. Both land near 9 GiB, but the 27B builds that fit a 12GB card are 2–3 bit and leave no room for context. Nobody has published a head-to-head at those quants, so treat any page that states one flatly with suspicion — including this one if it ever does.
Quick start
ollama run qwen3.5:9bhuggingface-cli download unsloth/Qwen3.5-9B-GGUF --include "*Q8_0*"ollama ps # PROCESSOR must read 100% GPUGo from "it runs" to actually building
All 561 chapters — running local models, RAG, agents, fine-tuning — plus the Python Lab and every course added later.
Frequently asked questions
Which Qwen model is best for an RTX 3060 12GB?
Qwen3.5-9B. At Q4_K_M it is 5.29 GiB, which on a 12GB card means you can step up to Q8_0 (8.87 GiB) instead of running 4-bit weights, and because only 8 of its 32 layers keep a KV cache the context costs 32 KiB/token instead of the 160 KiB/token a Qwen 3 14B charges. One pull: ollama pull qwen3.5:9b.
Qwen3.5-9B or Qwen 3 14B on 12GB?
The 14B has more parameters; the 9B has a newer architecture, image input, a 262K native context and five times less KV cache per token. On 12GB the 14B at Q4 (8.38 GiB) runs out of room somewhere around 16K of context, while the 9B at Q4 (5.29 GiB) still has headroom at 128K. If your prompts are short, the 14B is a fair fight. If they are not, it is not close.
Can 12GB VRAM run Qwen3.6 or Qwen3.8 27B?
Not usefully. Ollama lists qwen3.6:27b at 17GB and qwen3.8:27b at 18GB. In the Unsloth GGUF repo the smallest Qwen3.8-27B builds that come in under 12 GiB are UD-Q2_K_XL (9.15 GiB) and UD-IQ3_S (11.21 GiB) — 2-bit and low-3-bit, with nothing left for context. The 3-bit build people usually recommend, UD-Q3_K_XL, is 12.24 GiB and already over the line.
What is the best Qwen coding model for 12GB?
Qwen 2.5 Coder 14B at Q4_K_M — 8.37 GiB, and still the biggest Qwen coder that fits, because the 2026 Qwen coders start at 30B (Ollama lists qwen3-coder:30b at 19GB). Budget for its KV cache: 48 layers × 2 × 8 KV heads × 128 head_dim × 2 bytes is 192 KiB per token, so a 32K window would want 6 GiB on its own.
Related guides
Ready to Go Beyond Tutorials?
20 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?