AMD RX 9070 XT for Local AI: ROCm Setup, Real tok/s, and 16GB Model Picks

The RX 9070 XT became a genuinely good local AI card the day ROCm 7 made RDNA 4 official. Expect ~92 tok/s on gpt-oss:20b and ~52 tok/s on Qwen3 14B through Ollama — measured, cited below — with native gfx1201 support and zero workarounds. The one number that rules everything: 16GB. Stay at 9B-20B models and this $599-MSRP card outruns an RTX 5060 Ti 16GB by a wide margin; pull a dense 27B and throughput collapses to ~6 tok/s.

This card has a timing problem that most guides ignore. It launched in March 2025 as a gaming flagship with no real AI story — ROCm didn't support it, vLLM silently fell back to FP32, and every forum thread said "use Vulkan and pray." Most of that advice is still ranking, and most of it is now wrong. As of August 2026, the RX 9070 XT (gfx1201) sits on AMD's official ROCm supported-GPU list and in Ollama's supported-cards table. What follows is the current state: verified specs, third-party benchmarks with sources, the Vulkan-vs-ROCm split that actually holds, exact setup commands, and the limits nobody puts in a headline.

---

The Card, in AI Terms

One spec does most of the work: 16GB of GDDR6 at 640 GB/s. Bandwidth is what generation speed scales with, and 640 GB/s is 43% more than the RTX 5060 Ti 16GB's 448 GB/s — while the 16GB capacity sets a hard ceiling at roughly 20B-parameter models in 4-bit.

Specs per AMD's published figures:

A note on what it costs today: when we checked Newegg in the first week of August 2026, new cards started at $739.99 (XFX Swift) with most models between $740 and $800 — some 23-34% over the $599 MSRP. That is the memory-shortage market at work (background in our GPU prices and the memory shortage breakdown), though AMD's mid-range has inflated far less than NVIDIA's high end. Street prices move weekly, so check a live RX 9070 XT listing before doing any value math; every conclusion below holds at MSRP and weakens as the premium grows.

---

ROCm 7 Changed the Answer

RDNA 4 is now a first-class ROCm citizen. AMD's ROCm 7 system-requirements page lists the RX 9070 XT as officially supported on Ubuntu 24.04.4 / 22.04.5 (and RHEL 9.7 / 10.1), and Ollama's GPU docs list it natively — no HSA_OVERRIDE_GFX_VERSION, no community forks, no prayer.

The history matters because the internet hasn't caught up. In March 2026, Ivan Angelov's Vulkan-vs-ROCm benchmark on digtvbg.com found vLLM on ROCm 7.2 managing just 48 tok/s on a Qwen3.5 9B Q6 model while llama.cpp's Vulkan backend did 62 tok/s on identical hardware — because vLLM had no native gfx1201 kernels yet and was silently dequantizing to FP32, bypassing the card's 128 AI accelerators entirely. His conclusion at the time — "use Vulkan" — was right then. It is stale now for the llama.cpp/Ollama stack (current numbers in the next section), and it remains a fair warning for vLLM: verify kernel support before building a serving stack on this card.

Two platform notes worth knowing before you buy:

• Linux is the first-class path. Ollama's docs require the ROCm v7 driver on Linux and list the 9070 XT there; the Windows support table currently stops at RDNA 3.
• Windows still works — via Vulkan. llama.cpp's Vulkan backend needs nothing beyond the graphics driver, and one of the benchmark sources below ran its entire Windows test matrix that way (with ROCm on Windows available through pip install rocm-sdk for llama.cpp builds).

For the broader AMD stack — kernel modules, groups, PyTorch wheels, the works — our AMD ROCm setup guide is the long-form reference; this page keeps to what is 9070 XT-specific.

---

Real Benchmarks, Cited

Through Ollama on ROCm, a 9070 XT measures 91.9 tok/s on gpt-oss:20b, 57.8 tok/s on Qwen3.5 9B, 52.2 tok/s on Qwen3 14B — and 6.3 tok/s on a 27B that doesn't fit. None of these are our numbers; all three sources are public and linked.

Source 1 — Ollama + ROCm (Docker), Ubuntu 24.04. The hirokuze/local-llm-benchmark-rx9070xt suite ran a 43-question benchmark (speed + math/coding/MMLU/logic scoring) on a 9070 XT in a Ryzen 7 9800X3D system, using the official ollama/ollama:rocm image, v0.18.2, temperature 0.1, warmup per model:

That last row is the most useful number on this page. The 27B file is ~17GB; 16GB of VRAM means part of it lives in system RAM, and throughput falls off a cliff — 14.6x slower than gpt-oss:20b. The author's conclusion, which matches everything else we've seen on 16GB cards: keep models under ~20B at Q4.

Source 2 — llama.cpp b10068, ROCm vs Vulkan, Windows. The bhat-sahab/llama-inference repo publishes both backends on the same card (i9-14900K host). PP = prompt processing t/s, gen = generation tok/s:

Source 3 — llama.cpp upstream. A July 2026 llama.cpp issue from a 9070 XT owner documents Vulkan prefill on gpt-oss-20b Q4_K_M at ~3,450-3,650 t/s — and a 3-8% MoE prompt-processing regression on 64-CU RDNA 4 since a June 29 submission-heuristic change, still open at this writing. Two takeaways: prefill on this card is genuinely fast, and the software is still being actively tuned, so re-benchmark when you update.

Cross-check — the official llama.cpp performance threads. Community llama-bench submissions on the classic Llama 2 7B Q4_0 test show the same shape: ~137 tok/s generation under Vulkan vs ~101 under ROCm/HIP, with prefill around 5,000 t/s on both — about 42% above an RX 7900 XTX's prefill in the same thread, while the XTX generates ~35% faster (167-191 tok/s) on its 960 GB/s bus. Different rigs and commits, so treat the rows as indicative, but the prefill-monster / bandwidth-limited profile of this card is consistent everywhere we looked.

For calibration: chat feels instant above roughly 15-20 tok/s. Everything in the fits-in-VRAM rows is two to six times past that bar.

---

Vulkan vs ROCm

The measured split is consistent across sources: ROCm wins prompt processing by ~40-45%, Vulkan wins generation by ~12-18%. Ollama uses ROCm and is the right default; llama.cpp lets you pick per workload.

Which means the backend question is really a workload question:

• Long documents, RAG, big system prompts → you are prefill-bound → ROCm. 3,609 vs 2,490 t/s prefill is the difference between a 30-second and a 45-second first token on a very long context.
• Chat and coding assistants → you are generation-bound → Vulkan, worth ~10 extra tok/s on 9B-12B models. It is also the zero-install option: the graphics driver is all you need.
• Can't decide → run Ollama (ROCm) and stop thinking about it. The gap in either direction is real but not life-changing for interactive use.

The interesting outlier is the spilled MoE row above: once Qwen3-Coder-30B exceeds 16GB, ROCm generates faster (30.6 vs 22.6) while Vulkan prefills faster — backend behavior inverts when PCIe transfers enter the picture. If you plan to live in that partial-offload zone, test both on your exact model.

---

Setup in ~20 Minutes

On Ubuntu 24.04 it is three moves: install the amdgpu/ROCm stack, add your user to the render and video groups, install Ollama. gfx1201 is auto-detected — no environment variables.

Commands as published in AMD's ROCm quick-start at this writing (the version path changes; check repo.radeon.com for the current one — any ROCm 7.x supports gfx1201):

# 1. AMD GPU repo + kernel driver + ROCm (Ubuntu 24.04 "noble")
wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms rocm

# 2. GPU access for your user, then reboot
sudo usermod -a -G render,video $LOGNAME
sudo reboot

Verify the card is visible to ROCm:

rocminfo | grep -i gfx
# expect: gfx1201

Then Ollama — the installer detects ROCm on its own:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:20b

Prefer containers? The exact configuration the Source-1 benchmarks used is the official ROCm image with the two device nodes passed through:

docker run -d --name ollama \
  --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama -p 11434:11434 \
  ollama/ollama:rocm

Watch rocm-smi in a second terminal during your first generation — GPU utilization near 99% confirms nothing silently fell back to CPU. If you're new to Ollama itself (models, Modelfiles, the API), start with the complete Ollama guide.

One model-sourcing gotcha from the benchmark repo worth repeating: some raw Hugging Face GGUFs of the newest Qwen releases embed an architecture tag older Ollama builds don't recognize (unknown model architecture: 'qwen35'). Pulling the same model from the official Ollama registry (ollama pull qwen3.5:9b) sidesteps it.

---

What to Run in 16GB

The sweet spot is 9B-20B at Q4-Q6, and the headline pick is gpt-oss:20b: in the cited suite it tied the best accuracy score while running 76% faster than the dense 14B. Do not pull a dense 27B — that's the cliff.

The short list, every number from the benchmarks above:

• gpt-oss:20b — the default. MoE architecture means 20B-class quality at 91.9 tok/s, and it fits with room for context. If you install one model, install this one.
• qwen3:14b — same 33/34 accuracy in the suite, half the speed (52.2 tok/s). Worth it when you want a dense model's steadier long-form output.
• gemma-4-12b (Q4_K_M, 6.7GB) — 54-65 tok/s depending on backend; leaves ~9GB free, which is the comfortable choice if you also want big context windows or a second resident model.
• qwen3.5:9b — 57.8-92.3 tok/s depending on backend and quant; the speed pick, with the caveat that it scored notably lower on knowledge-heavy questions in the cited suite (25/34).
• Qwen3-Coder-30B-A3B — the one justified over-budget model. ~18GB, spills past VRAM, yet still generates ~30 tok/s under ROCm because only ~3B parameters are active per token. Fine for conversational coding; painful for long-context work (prefill drops to hundreds, not thousands, of t/s).

The anti-pick: any dense 27B+ at Q4. A ~17GB file on a 16GB card ran at 6.3 tok/s in testing — a minute-plus per paragraph. If 27B-32B models are the actual goal, buy 24GB instead (next section).

Our tier pages go deeper on this class of card: best LLMs for 16GB VRAM for the general list, best coding LLMs for 16GB if the card's job is programming, and the Ollama RAM/VRAM table for checking any specific model tag before you pull it.

---

vs RTX 5060 Ti 16GB

The 9070 XT is the faster 16GB card — its 43% bandwidth advantage shows up almost linearly in tok/s (~52 vs ~33 on 14B Q4). The 5060 Ti 16GB is $170 cheaper at MSRP, pulls 124W less, and buys you CUDA. That last word decides most purchases.

9070 XT figures from the cited third-party benchmarks; 5060 Ti figures from our own 5060 Ti 16GB review.

Choose the 9070 XT if you run Linux, want the fastest 16GB inference available, or the card is doing double duty as a high-end gaming GPU. Choose the 5060 Ti 16GB if you need the CUDA ecosystem — ExLlamaV2, TensorRT-LLM, mainstream fine-tuning — or a low-power small-form-factor build.

And the honest third option: spend more, get 24GB. AMD's own RX 7900 XTX runs 32B-class models a 16GB card cannot hold at any price, and the used market has its own math — see the used GPU buying guide. The full cross-vendor landscape, Intel included, is in AMD vs NVIDIA vs Intel for AI.

---

Honest Limitations

Buy this card knowing four things: 16GB is a hard ceiling, CUDA-only tools are permanently off the menu, Windows support is thinner than Linux, and parts of the stack are still settling.

1. The 16GB ceiling is absolute for dense models. No setting fixes 6.3 tok/s on a spilled 27B. The 9070 XT is a 20B-and-under card; 24GB cards are a different class.
2. No CUDA. ExLlamaV2, TensorRT-LLM, and most fine-tuning tutorials assume NVIDIA. Inference on AMD is now genuinely good; training and the long tail of CUDA-kernel tooling are not.
3. vLLM needs verification before you commit. As late as March 2026 it lacked native gfx1201 kernels and quietly ran FP32 dequantization (the digtvbg finding). Check current vLLM-ROCm release notes before building a serving stack on this card; llama.cpp and Ollama are the proven paths today.
4. Windows is second-class. Ollama's docs list the card for Linux only. llama.cpp Vulkan covers Windows well — but if your workflow is Ollama-centric, plan on Linux.
5. The stack is still moving. The open llama.cpp MoE-prefill regression (3-8% on 64-CU RDNA 4) is small, but it's a reminder: numbers here were true at the linked commits, and this platform improves — and occasionally regresses — month to month.
6. You probably won't pay $599. Shortage pricing has most cards above MSRP; the market context explains why, and the value verdict tightens with every dollar over list.

---

Verdict

At MSRP, the RX 9070 XT is the best-performing 16GB card for local AI you can buy, and ROCm 7 removed the asterisk that used to follow that sentence. ~92 tok/s on a 20B-class MoE is genuinely quick — past the instant-feeling threshold with a 4-6x margin — and setup on Ubuntu is three commands and a reboot.

The buying logic in one paragraph: if your models are 9B-20B and your OS is Linux, this card at a sane price beats everything else with 16GB. If you need CUDA or sip power, the 5060 Ti 16GB gives up speed to get both. If your ambitions say 27B or more, no 16GB card is the answer — start at 24GB. And in this market, always check the street price before trusting anyone's value verdict, including ours: current listings.

---

Sources

• AMD ROCm system requirements — official RX 9070 XT (gfx1201) support and OS matrix
• ROCm Linux quick-start — install commands reproduced above
• Ollama GPU documentation — supported AMD cards and driver requirements
• hirokuze/local-llm-benchmark-rx9070xt — Ollama/ROCm tok/s + accuracy suite (Source 1)
• bhat-sahab/llama-inference — llama.cpp b10068 ROCm-vs-Vulkan tables (Source 2)
• llama.cpp issue #26346 — RDNA 4 Vulkan prefill data and MoE regression (Source 3)
• Ivan Angelov, digtvbg.com — March 2026 vLLM/Vulkan state of play
• llama.cpp community performance threads — ROCm/HIP #15021 and Vulkan #10879 llama-bench submissions (cross-check)
• AMD RX 9070 XT published specifications — card specs table (cross-checked against Sapphire and ASUS board-partner spec sheets)
• Newegg listings, first week of August 2026 — street pricing for the RX 9070 XT and RTX 5060 Ti 16GB

---

FAQ