Can I Run AI on Ubuntu? Ollama, CUDA & ROCm Setup

Published June 20, 2026 • 11 min read

Yes — Ubuntu runs local AI, and it is arguably the best operating system for it. Ubuntu installs the most popular local-AI runtime, Ollama, in one command (curl -fsSL https://ollama.com/install.sh | sh), runs entirely offline, and supports both NVIDIA (CUDA) and AMD (ROCm) GPU acceleration natively. A machine with 16GB of RAM and no GPU can run a 3B–8B model on CPU; a 24GB GPU has room for a 32B model at 4-bit quantization. Ubuntu 22.04, 24.04, and 26.04 LTS share the same installer and service layout, so the steps below work on every supported release.

Most "run AI on Ubuntu" guides skip the part that actually decides your experience: whether the model you want fits, and how fast it can possibly go on the hardware you have. This page gives you the commands, the driver setup for both GPU vendors, and — instead of numbers you have to take on faith — the two pieces of arithmetic that let you work out the answer for your own machine before you download anything.

Can Ubuntu actually run local AI models?

Yes. Ubuntu is the reference platform for most of the local-AI ecosystem. Ollama, llama.cpp, vLLM, and Hugging Face Transformers all ship first-class Linux support and are frequently developed and tested on Ubuntu first. The official Ollama Linux installer creates a systemd service, binds the local API to 127.0.0.1:11434, and auto-detects your GPU — no cloud account, no API key, and no data ever leaves your machine.

Three things make Ubuntu a strong AI host:

• One-command install. No package juggling — the upstream script handles everything, including the GPU runtime.
• Native GPU acceleration. Ollama bundles its own CUDA runtime for NVIDIA cards, so you only install the driver — not the full CUDA Toolkit. AMD GPU acceleration via ROCm is Linux-only, and Ubuntu is its best-supported home.
• Lower overhead than Windows. A headless Ubuntu Server install leaves more RAM and VRAM free for the model than a desktop OS.

How do I install AI on Ubuntu? (the 3-command version)

The fastest path to running AI locally on Ubuntu is Ollama. On a fresh Ubuntu 22.04, 24.04, or 26.04 system:

# 1. Install Ollama (installs the systemd service + GPU runtime)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Confirm it installed
ollama --version

# 3. Pull and run your first model
ollama run llama3.1:8b "Explain what a transformer model is in two sentences."

That is genuinely it. The first run downloads the model (Llama 3.1 8B is about 4.9GB at the default Q4 quantization), then drops you into an interactive prompt. If curl is missing, install it first with sudo apt update && sudo apt install -y curl.

Almost all of the wait is the download, not the setup. You can work out your own wait before you start: minutes = file size in GB × 8 ÷ (your Mbps) × 1000 ÷ 60. A 4.9GB model on a 100 Mbps line is 4.9 × 8 = 39.2 gigabits ÷ 0.1 Gbps ≈ 392 seconds, so about six and a half minutes at full line speed. On 25 Mbps, budget half an hour.

Do I need a GPU, or will CPU-only work?

You do not need a GPU. Ollama runs every model on CPU if no supported GPU is found — it is just slower, and the reason is worth understanding because it also tells you what to buy.

Generating one token requires reading the entire model's weights out of memory. So the hard ceiling on generation speed is a division:

tokens/second ceiling = memory bandwidth (GB/s) ÷ model size in memory (GB)

That is an arithmetic upper bound, not a promise — real output lands well below it, because attention over a growing context, sampling and framework overhead all cost time the formula ignores. But it explains the entire CPU-versus-GPU gap in one line: dual-channel DDR5 system RAM moves on the order of 80–90 GB/s, while a mid-range consumer GPU moves several hundred GB/s and a 3090-class card is near 1 TB/s (see NVIDIA's published card specifications for the exact figure for your model). Same model, roughly an order of magnitude difference in the ceiling.

The second piece of arithmetic decides whether the model fits at all. The model's quantized file size must fit in your VRAM, plus headroom for the KV cache and context. For the Q4_K_M quantization Ollama ships by default, a good estimate is:

VRAM needed (GB) ≈ 0.6 × parameters in billions

So an 8B model is ~4.8GB, a 14B is ~8.4GB, a 27B is ~16GB and a 32B is ~19GB — then add roughly 1–2GB for an 8K context. Compare that to the file size Ollama reports on the model's library page before you pull it; the two should land within a few hundred megabytes of each other.

NVIDIA (CUDA) vs AMD (ROCm): how do I set up the driver?

NVIDIA — CUDA

You only need the NVIDIA driver. Ollama ships its own CUDA runtime, so skip the full CUDA Toolkit unless you are compiling other AI software.

# See which driver Ubuntu recommends for your card
ubuntu-drivers devices

# Install the recommended branch (works on 22.04, 24.04, and 26.04)
sudo ubuntu-drivers install
sudo reboot

# After reboot, confirm the GPU is visible
nvidia-smi

(On Ubuntu 22.04 and 24.04 the older sudo ubuntu-drivers autoinstall still works as a deprecated alias, but install is the current command and is the one that exists on 26.04.)

If nvidia-smi prints your GPU, driver version, and CUDA version without errors, Ollama will use the GPU automatically — no extra config. (On recent Ubuntu LTS releases the proprietary driver ships in the restricted repository component, so the default repo is usually all you need.)

AMD — ROCm

AMD acceleration in Ollama is Linux-only, and Ubuntu is the best-supported platform. RDNA 3 cards (RX 7900 XTX, 7900 XT, 7800 XT) have solid support.

The amdgpu-install helper is not in Ubuntu's default repositories — you grab the small .deb from AMD's ROCm repo first, then run it. Browse AMD's ROCm repo to copy the exact current filename for your Ubuntu codename (noble = 24.04, jammy = 22.04); the example below uses a recent build for 24.04.

# 1. Download AMD's installer .deb (grab the current filename from the repo above)
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/noble/amdgpu-install_7.2.3.70203-1_all.deb
sudo apt install -y ./amdgpu-install_7.2.3.70203-1_all.deb

# 2. Install the ROCm use case
sudo amdgpu-install --usecase=rocm

# 3. Give your user GPU access, then reboot
sudo usermod -aG render,video $USER
sudo reboot

Ollama auto-detects the ROCm environment after that — no extra flags needed. For Ollama specifically, NVIDIA still has the software-maturity edge (wider model compatibility and faster driver cadence), but AMD on Linux with RDNA 3 now delivers competitive performance at a better price-per-GB-of-VRAM.

Which models can I run on my hardware? (the tier table)

This is the question that actually decides your experience. The table below applies the two formulas above to common hardware tiers. VRAM figures are for Q4_K_M quantization, the default Ollama ships for most models. Bandwidth figures are the manufacturers' published specifications; the last column is bandwidth ÷ model size, so it is a ceiling you will not reach, not an expected result — but it is honest arithmetic you can redo for any card.

Bandwidth from each card's specification page. Ceiling = bandwidth ÷ model size, per the formula above.

Two things in that table are more useful than any single number:

• The CPU-only row is not disqualifying, but it is misleading in your favour. The bandwidth ceiling looks respectable; CPUs fall much further below their ceiling than GPUs do, because they are also short of the raw arithmetic throughput to keep up. Treat CPU-only as "usable for background work and patient chat", not as a GPU substitute.
• Bandwidth, not VRAM size, sets the speed. The 16GB RTX 4060 Ti has more memory than a 3060 12GB and a lower ceiling on the same model, because its bus is narrower. This is why "how much VRAM" is only half the buying question — check the bandwidth number on the spec sheet too.

And the practical read on capacity:

• 8GB is the sweet spot for getting started. Llama 3.1 8B (~4.9GB as shipped) is one of the most widely used local models and still leaves room for a healthy context window on an 8GB card.
• 16GB lets you step up to a 14B without paying for a 24GB card. Qwen2.5 14B is a strong general/coding pick here.
• 24GB unlocks 32B-class models, which is where local quality starts feeling genuinely close to small cloud models for many tasks.

Want your real number rather than the ceiling? Ollama prints it for you — run any prompt with ollama run llama3.1:8b --verbose and read the eval rate line it prints after the response. That is a measurement of your machine, which beats anyone else's.

How do I pick and pull the right model?

Match the model to your GPU tier above, then pull it. A few commands to copy:

# CPU-only or 8GB GPU — fast, capable all-rounder
ollama pull llama3.1:8b

# 8GB GPU — strong multilingual / coding option
ollama pull qwen2.5:7b

# 16GB GPU — step up to 14B
ollama pull qwen2.5:14b

# 24GB GPU — 32B-class quality
ollama pull qwen2.5:32b

Want a guided recommendation based on your exact CPU, RAM, and GPU? Run your specs through our "Can I run local AI?" checker — it tells you which models will fit and how fast they should go before you download anything.

Can I use these models from my own apps and IDE?

Yes. Once Ollama is running, it exposes a local HTTP API on port 11434 that most AI tooling speaks:

# Call the local model over HTTP — no internet required
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write a Python function to reverse a string.",
  "stream": false
}'

From there you can wire it into VS Code (via the Continue extension), JetBrains IDEs, a RAG pipeline, or your own Python/Node scripts — all pointed at localhost, all fully offline. The official documentation lives in the Ollama GitHub repository and the Ollama model library.

Key Takeaways

1. Yes, Ubuntu runs AI — exceptionally well. It is the reference platform for the local-AI ecosystem, and the same installer works on Ubuntu 22.04, 24.04, and 26.04 LTS.
2. Install is one command. curl -fsSL https://ollama.com/install.sh | sh, then ollama run llama3.1:8b.
3. No GPU is required — CPU works, just slower. A GPU mainly buys responsiveness and the ability to run larger models.
4. NVIDIA needs only the driver (Ollama bundles CUDA); AMD uses ROCm, which is Linux-only and well-supported on Ubuntu.
5. Size the model with arithmetic, not guesswork: VRAM ≈ 0.6GB per billion parameters at Q4_K_M — ~4.8GB for an 8B, ~8.4GB for a 14B, ~19GB for a 32B, plus 1–2GB for context.
6. Speed is capped by memory bandwidth ÷ model size. Check the bandwidth on your card's spec sheet, then measure the real figure with ollama run --verbose.

Next Steps

• New to Ollama itself? Start with our complete Ollama guide for every command, flag, and Modelfile option.
• On a modest machine? See the best local AI models for 8GB of RAM for picks that stay fast without a high-end GPU.
• Not sure if your exact rig is enough? Use the "Can I run local AI?" hardware checker to get a model recommendation in seconds.
• Shopping for an upgrade? Our best GPUs for AI guide breaks down VRAM-per-dollar across NVIDIA and AMD.