AMD ROCm Setup for Local LLMs (2026): Radeon, Strix Halo, MI300X

AMD ROCm has quietly become a credible alternative to NVIDIA CUDA for local LLMs. A Radeon RX 7900 XTX runs Llama 3.1 8B at ~96 tok/s — 75% of an RTX 4090's speed at less than half the price. Strix Halo (Ryzen AI Max+ 395) puts 128GB of unified memory in a 65W mini-PC, running 70B models entirely on the iGPU. And MI300X with 192GB HBM3 is outperforming H100 on long-context inference.

This guide is the complete reference: installing ROCm, configuring your GPU, running Ollama / llama.cpp / vLLM / PyTorch on AMD, FlashAttention support, real benchmarks vs CUDA, and the gotchas nobody tells you about. We cover Radeon RX 7000/9000-series, Strix Halo, and CDNA 3 (MI300X / MI325X).

Table of Contents

1. The State of ROCm in 2026
2. Supported AMD Hardware
3. ROCm vs CUDA: What's Actually Different
4. Installation: Ubuntu, Fedora, WSL2
5. Verifying Your Install
6. Ollama on ROCm
7. llama.cpp on ROCm
8. vLLM on ROCm
9. PyTorch on ROCm
10. FlashAttention on AMD
11. Quantization on ROCm
12. Tuning the Radeon RX 7900 XTX
13. Strix Halo / Ryzen AI Max+ 395
14. MI300X / MI325X for Servers
15. HSA_OVERRIDE_GFX_VERSION: Unofficial Cards
16. Performance Benchmarks
17. Troubleshooting
18. FAQ

---

The State of ROCm in 2026 {#state-of-rocm}

For two years AMD users were second-class citizens in local AI. That ended in 2024-2025. Today:

• Ollama, llama.cpp, vLLM, PyTorch, JAX, Triton, ExecuTorch all ship ROCm builds.
• FlashAttention-2 has an official AMD port for RDNA 3 and CDNA 3.
• Strix Halo / Ryzen AI Max+ put 128GB of unified memory on a desktop platform.
• MI300X (192GB HBM3) is in production at major hyperscalers and beats H100 on long-context inference.
• Radeon RX 7900 XTX at $750-900 hits 75% of RTX 4090 inference speed.

What still lags: training tooling parity, FP8 software maturity (works on MI300X, weak on RDNA 3), some xformers / SDXL paths, and Windows native support.

For pure inference, ROCm is a real choice in 2026. For training large models, NVIDIA still wins.

---

Supported AMD Hardware {#hardware}

Officially supported in ROCm 6.x

Unofficially supported (HSA_OVERRIDE_GFX_VERSION)

These work for inference on Ollama / llama.cpp but are unsupported for production. See HSA_OVERRIDE_GFX_VERSION below.

---

ROCm vs CUDA: What's Actually Different {#rocm-vs-cuda}

The good news: HIP is mostly source-compatible with CUDA. hipify translates CUDA code automatically, so most frameworks ship both backends from the same codebase.

The bad news: kernels hand-tuned for NVIDIA Tensor Cores (FlashAttention-3, fused MoE kernels, etc.) need separate AMD implementations and tend to lag.

---

Installation: Ubuntu, Fedora, WSL2 {#installation}

Ubuntu 22.04 / 24.04 (recommended)

# Add AMDGPU repo and install (ROCm 6.2)
wget https://repo.radeon.com/amdgpu-install/6.2/ubuntu/jammy/amdgpu-install_6.2.60200-1_all.deb
sudo apt install ./amdgpu-install_6.2.60200-1_all.deb
sudo amdgpu-install --usecase=rocm,hiplibsdk -y

# Add yourself to required groups
sudo usermod -aG render,video $USER

# Reboot
sudo reboot

For Ubuntu 24.04, replace jammy with noble in the URL.

Fedora 39+

sudo dnf install rocm-hip rocm-hip-devel rocm-comgr rocm-runtime
sudo usermod -aG render,video $USER
sudo reboot

WSL2 (Windows 10/11)

ROCm in WSL2 supports a narrower set of GPUs (RX 7900 XTX/XT/GRE, Pro W7900/W7800, Strix Halo) with the AMD Software: Adrenalin Edition for WSL driver.

# Inside WSL2 Ubuntu 22.04
wget https://repo.radeon.com/amdgpu-install/6.2/ubuntu/jammy/amdgpu-install_6.2.60200-1_all.deb
sudo apt install ./amdgpu-install*.deb
sudo amdgpu-install -y --usecase=wsl,rocm --no-dkms

Critical: --no-dkms because WSL uses the Windows driver. Skip the Linux kernel module install.

Docker

docker run -it --rm \
    --device /dev/kfd --device /dev/dri \
    --group-add video --group-add render \
    --security-opt seccomp=unconfined \
    rocm/dev-ubuntu-22.04:6.2 bash

The /dev/kfd (Kernel Fusion Driver) and /dev/dri devices are how ROCm reaches the GPU.

---

Verifying Your Install {#verify}

# Should list your GPU
rocminfo | grep -A 5 "Agent"

# GPU utilization, temp, power
rocm-smi

# Detailed
rocm-smi --showallinfo

Expected output for a Radeon RX 7900 XTX:

Agent 2
*******
  Name: gfx1100
  Marketing Name: Radeon RX 7900 XTX
  Vendor Name: AMD
  Feature: KERNEL_DISPATCH
  ...

If you see only gfx000 or no agents, your driver did not load — check dmesg | grep amdgpu.

---

Ollama on ROCm {#ollama}

Ollama ships a ROCm binary that auto-detects the GPU.

curl -fsSL https://ollama.com/install.sh | sh

The installer prints the detected backend. For unofficially-supported GPUs:

# RX 6800 / 6900 XT
HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve

# Strix Halo iGPU
HSA_OVERRIDE_GFX_VERSION=11.5.1 ollama serve

# Set persistently in systemd
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf <<EOF
[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.5.1"
Environment="HCC_AMDGPU_TARGET=gfx1151"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify GPU is being used:

ollama run llama3.1:8b "hi"
# In another terminal:
rocm-smi
# Should show >0% GPU utilization during inference

---

llama.cpp on ROCm {#llamacpp}

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build with HIP backend
HIPCXX="$(hipconfig -l)/clang" \
HIP_PATH="$(hipconfig -R)" \
cmake -B build \
    -DGGML_HIP=ON \
    -DAMDGPU_TARGETS=gfx1100 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

./build/bin/llama-cli -m model.gguf -ngl 999 -fa

AMDGPU_TARGETS should match your GPU's gfx version. For multiple GPUs, use gfx1100;gfx1101.

llama.cpp Vulkan (alternative)

For unsupported AMD GPUs, the Vulkan backend works on anything with Vulkan 1.2:

cmake -B build-vk -DGGML_VULKAN=ON
cmake --build build-vk -j

Vulkan is 60-80% the speed of HIP/ROCm but works on RX 5000-series, Intel Arc, and even some integrated GPUs. See our Intel Arc A770 guide for similar patterns.

---

vLLM on ROCm {#vllm}

docker pull rocm/vllm:latest

docker run --device /dev/kfd --device /dev/dri \
    --group-add video --group-add render \
    --security-opt seccomp=unconfined \
    --shm-size 16G \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    rocm/vllm:latest \
    vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 16384

For supported AWQ quantization on RDNA 3:

vllm serve casperhansen/llama-3.1-8b-instruct-awq \
    --quantization awq \
    --max-model-len 16384

FP8 weights work on MI300X/MI325X but not on RDNA 3. INT8 (W8A8) works on both.

For full vLLM tuning, see our vLLM Complete Setup Guide — most flags are identical between CUDA and ROCm.

---

PyTorch on ROCm {#pytorch}

# Stable wheel for ROCm 6.2
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.2

# Verify
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

torch.cuda.is_available() returns True even on AMD because the HIP runtime maps to the CUDA API. torch.version.hip returns the ROCm version.

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Most Hugging Face models work unchanged. Exceptions: anything that imports xformers (use attn_implementation="sdpa" instead) or bitsandbytes (use AWQ / GPTQ instead, or the bitsandbytes-rocm fork).

---

FlashAttention on AMD {#flash-attention}

RDNA 3 (RX 7900-series)

git clone https://github.com/ROCm/flash-attention
cd flash-attention
GPU_ARCHS="gfx1100" python setup.py install

This is AMD's fork; the upstream Tri Dao FlashAttention-2 also has CK (Composable Kernel) support but the AMD fork is generally faster on RDNA 3.

CDNA 3 (MI300X)

pip install flash-attn --no-build-isolation

Upstream FlashAttention-2 supports MI300X via CK kernels in v2.5+.

Use in llama.cpp / vLLM / PyTorch

• llama.cpp: -fa flag (auto-detected)
• vLLM: auto-detected
• PyTorch: torch.nn.functional.scaled_dot_product_attention automatically uses FlashAttention when shapes are favorable

Performance impact

Same pattern as NVIDIA — FlashAttention is mandatory for long context.

---

Quantization on ROCm {#quantization}

For RDNA 3, AWQ-INT4 with vLLM or Q5_K_M / Q4_K_M with llama.cpp are the practical defaults. For MI300X, FP8 + AWQ-INT4 + INT8 W8A8 all work.

See AWQ vs GPTQ vs GGUF for the underlying theory.

---

Tuning the Radeon RX 7900 XTX {#radeon-7900}

Power and thermals

# Set power cap to 290W (stock 355W) — saves ~15% power, ~3% perf loss
sudo rocm-smi --setpoweroverdrive 290

# Lock GPU clock for predictable latency
sudo rocm-smi --setperflevel high

# Fan curve
sudo rocm-smi --setfan 70%

llama.cpp recipe

./build/bin/llama-server \
    -m llama-3.1-8b-instruct-q5_k_m.gguf \
    -ngl 999 \
    -c 16384 \
    -fa \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    -t 8 \
    --host 0.0.0.0 --port 8080

vLLM recipe

vllm serve casperhansen/llama-3.1-8b-instruct-awq \
    --quantization awq \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-num-batched-tokens 8192

Multi-GPU 7900 XTX

# llama.cpp tensor split for 2x 7900 XTX
./llama-cli -m 70b-q4.gguf -ngl 999 --tensor-split 24,24

# vLLM TP=2
vllm serve <70b-awq-model> --tensor-parallel-size 2 --quantization awq

7900 XTX does not support NVLink/Infinity Fabric Link, so multi-GPU is PCIe-only. PCIe 4.0 x16 (~32 GB/s practical) is the bottleneck for tensor parallelism — expect 1.5-1.7x speedup with TP=2 vs single-GPU on a 70B model that fits split.

---

Strix Halo / Ryzen AI Max+ 395 {#strix-halo}

The most interesting AMD platform of 2025-2026 for local AI.

Hardware

• CPU: Zen 5, 16 cores / 32 threads
• GPU: RDNA 3.5, 40 CUs, gfx1151
• Memory: Up to 128 GB LPDDR5X-8000, ~256 GB/s bandwidth
• NPU: XDNA 2, ~50 TOPS (separate accelerator)
• Power: 55-120W configurable
• Form factor: Mini-PC (Framework Desktop, Asus, HP), laptops

Why it matters

128 GB of unified memory means a 70B model in BF16 fits entirely on the iGPU — no quantization required for quality-critical workloads. No consumer NVIDIA card has this much memory. The closest comparison is a Mac Studio M4 Max with 128 GB unified at $4,000+; Strix Halo systems start ~$2,000.

Setup

# Override gfx version
echo 'export HSA_OVERRIDE_GFX_VERSION=11.5.1' >> ~/.bashrc
echo 'export HCC_AMDGPU_TARGET=gfx1151' >> ~/.bashrc
source ~/.bashrc

# Install ROCm 6.3+ (Strix Halo support added)
sudo amdgpu-install --usecase=rocm,hiplibsdk

# Reserve memory for iGPU (BIOS or sysctl)
# Most Strix Halo systems let you allocate 96-110 GB to GPU in BIOS

Performance (Llama 3.1 70B)

For comparison: 2x RTX 4090 (PCIe, no NVLink) on the same 70B Q4 hits ~24 tok/s — at much higher cost and power. Strix Halo is the best "70B in a quiet desktop" option in 2026 below Mac Studio prices.

Caveats

• LPDDR5X bandwidth (~256 GB/s) is much lower than discrete GPU VRAM (1,000+ GB/s), so single-stream throughput on small models that fit in 24 GB lags discrete GPUs significantly.
• GPU and CPU share the memory bus — heavy CPU workloads steal bandwidth from inference.
• Long prompt prefill is slow vs discrete GPUs because compute is bandwidth-bound on this platform.

---

MI300X / MI325X for Servers {#mi300x}

Specs

Why MI300X wins on certain LLM workloads

LLM inference is memory-bandwidth-bound for decode and capacity-bound for long-context KV-cache. MI300X's 192 GB at 5.3 TB/s beats an H100's 80 GB at 3.35 TB/s on both axes. Real-world wins:

• Llama 3.1 405B FP8 on a single MI300X — fits with room for a 32K KV cache. H100 needs at least 4-way TP.
• Long-context (128K) inference — MI300X holds the KV cache without offload at large batch sizes.
• Multi-tenant servers — more concurrent requests fit per GPU.

vLLM on MI300X

docker run --device=/dev/kfd --device=/dev/dri \
    --security-opt seccomp=unconfined --shm-size 32G \
    --network=host \
    rocm/vllm:latest \
    vllm serve meta-llama/Llama-3.1-405B-Instruct-FP8 \
    --quantization fp8 \
    --kv-cache-dtype fp8_e4m3 \
    --tensor-parallel-size 1 \
    --max-model-len 65536 \
    --gpu-memory-utilization 0.93 \
    --enable-prefix-caching

For multi-GPU MI300X (8x in OAM), use --tensor-parallel-size 8 and ensure RCCL is configured:

export NCCL_DEBUG=WARN     # RCCL respects NCCL_* vars
export RCCL_MSCCL_ENABLE=1 # MSCCL collective optimizations

---

HSA_OVERRIDE_GFX_VERSION: Unofficial Cards {#hsa-override}

Many older Radeon cards work with ROCm by lying about their gfx version. ROCm ships kernels for officially-supported gfx versions; unsupported cards run those kernels by claiming a compatible architecture.

Set globally:

echo 'export HSA_OVERRIDE_GFX_VERSION=10.3.0' >> ~/.bashrc

Or per-application via systemd override (see Ollama section).

Risks: unsupported configurations can crash, hang, or silently produce wrong results on edge-case operations. For inference of standard Llama / Mistral / Qwen models the failure modes are usually clean (crashes, not silent wrong outputs). Do not use HSA override for production.

---

Performance Benchmarks {#benchmarks}

All benchmarks: Ollama with Q4_K_M quantization at 4K context, single user, room-temperature 22°C ambient.

Llama 3.1 8B

Llama 3.1 70B Q4

MI300X is clearly the throughput leader for 70B. For the home / SMB segment, Strix Halo is genuinely competitive with multi-GPU rigs at lower cost and dramatically lower power.

---

Troubleshooting {#troubleshooting}

---

FAQ {#faq}

See answers to common AMD ROCm questions below.

---

Sources: AMD ROCm Documentation | Ollama AMD support | vLLM ROCm Docker | llama.cpp HIP build instructions | AMD Strix Halo product page | Internal benchmarks on RX 7900 XTX, Strix Halo, and MI300X.

Related guides on Local AI Master:

• AMD vs NVIDIA vs Intel AI GPU
• CUDA Optimization for Local LLMs
• vLLM Complete Setup Guide
• Best GPUs for AI 2025
• Linux Local AI Setup
• Complete Ollama Guide
• Intel Arc A770 Local AI