Docker Can't See Your GPU: Fix "could not select device driver" in Five Steps

Step 2: Install the NVIDIA Container Toolkit (not nvidia-docker2)

The package you want is nvidia-container-toolkit. NVIDIA's current install guide pins version 1.20.0-1.

The repository setup below is copied verbatim from NVIDIA's installation guide, which is also the exact sequence Ollama's Docker documentation links to. If you want the reproducible build, pin the version; if you want whatever is current, drop the pin and install the bare package name.

# Ubuntu / Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Fedora / RHEL / Amazon Linux
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
  | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit

# openSUSE / SLE
sudo zypper ar https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
sudo zypper --gpg-auto-import-keys install -y nvidia-container-toolkit

Source: NVIDIA Container Toolkit installation guide, section "Installation", as published on August 18, 2026.

Step 3: Register the runtime with Docker — the step everyone skips

Installing the package does not tell Docker about it. Until nvidia-ctk rewrites /etc/docker/daemon.json and the daemon restarts, --gpus all keeps failing with the same error.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

If you are running rootless Docker, this command writes to the wrong file. NVIDIA documents a separate three-command path: point nvidia-ctk at $HOME/.config/docker/daemon.json, restart with systemctl --user restart docker, then set nvidia-container-cli.no-cgroups in the toolkit config. Pick "rootless" in the picker above for the exact block.

Step 4: Run the canary before you touch your real stack

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi is NVIDIA's own sample workload. If it prints your GPU table, Docker is fixed and any remaining problem belongs to your application container.

Note the image: plain ubuntu. You do not need a CUDA image to test this, because the toolkit injects the driver libraries into whatever container you run. That matters, because half the guides on the internet hand you a nvidia/cuda: tag that was deleted years ago, and you end up debugging a manifest-not-found error instead of your actual problem. If you do want a CUDA base image, pull the tag from the current nvidia/cuda tag listing rather than copying one out of a tutorial — the tag set is pruned regularly, and a tag that worked when an article was written is the single most common reason these commands fail for a reason unrelated to your GPU.

# NVIDIA's documented sample workload
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

# Ollama's variant, if you prefer
docker run --gpus all ubuntu nvidia-smi

Sources: NVIDIA Container Toolkit "Running a Sample Workload"; Ollama troubleshooting docs, "Linux NVIDIA Troubleshooting".

Step 5: Translate it to Compose

Compose does not understand --gpus. You declare a device reservation, and the capabilities field is mandatory — omit it and the service errors on deployment.

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

count takes an integer or all. device_ids takes host GPU IDs as printed by nvidia-smi. Docker's docs state the two are mutually exclusive — set one, never both, or the service fails to start. Once this works, our Ollama + Open WebUI Compose walkthrough is the happy path from here.

The error strings, decoded

Five different messages, four different root causes. Match the exact text you pasted into Google against the left column. Every fix in the right column is from vendor documentation, cited below the table.

Sources: NVIDIA Container Toolkit installation guide and release notes (v1.20.0); Docker Docs, "Enable GPU access with Docker Compose"; Ollama troubleshooting documentation. Checked August 18, 2026. We have ordered these by how often they turn out to be the answer in support threads we read, not by any measured distribution — treat the ordering as a heuristic, not a statistic.

The Docker Desktop and WSL2 case (the most confused one)

On Docker Desktop you do not install the NVIDIA Container Toolkit inside your WSL distro. The GPU comes through WSL2 GPU paravirtualization, and everything you configure lives on the Windows side.

Docker's documentation lists four prerequisites: a Windows machine with an NVIDIA GPU, an up-to-date Windows 10 or 11 install, an up-to-date Windows NVIDIA driver that supports WSL2 GPU paravirtualization, and the latest WSL2 Linux kernel (wsl --update) — with the WSL2 backend turned on in Docker Desktop settings. Nothing in that list is a package inside Ubuntu-on-WSL.

The trap: people follow a Linux tutorial, apt-get install an NVIDIA driver inside the distro, and break the paravirtualized one that was already working. If you have done that, remove the Linux-side driver packages before anything else.

It worked, then it silently dropped to CPU

Add "exec-opts": ["native.cgroupdriver=cgroupfs"] to /etc/docker/daemon.json on the host and restart Docker.

This one deserves its own section because it does not look like a GPU problem. Nothing errors. The container keeps answering. Tokens just get slow, and the only visible clue is GPU-discovery failures in the container log. Ollama's troubleshooting page documents the cause — systemd cgroup management in Docker detaching the GPU from a long-running container — and this fix appears in essentially no general CUDA-in-Docker guide.

sudo nano /etc/docker/daemon.json
# merge this key with whatever is already in the file
#   "exec-opts": ["native.cgroupdriver=cgroupfs"]

sudo systemctl restart docker
docker logs -f ollama   # watch for GPU discovery lines on the next request

Before you blame the runtime, rule out the boring explanation: a model that no longer fits. Ollama offloads layers to CPU when VRAM runs short, and the symptom is identical. Check the numbers in our VRAM calculator, and see the Ollama troubleshooting guide for the non-Docker version of this diagnosis.

AMD and ROCm: forget --gpus entirely

ROCm containers take --device /dev/kfd --device /dev/dri and, on many distros, explicit --group-add flags with numeric group IDs read from the host.

If you typed --gpus all on an AMD box you will chase the device-driver error forever, because that flag is the NVIDIA path. Ollama's documented AMD invocation maps the two device nodes and uses the :rocm image tag. Access to /dev/kfd normally needs video and/or render group membership; inside a container you have to add those groups by their numeric ID, because the names do not map across the boundary.

# read the NUMERIC gids (note the -n)
ls -lnd /dev/kfd /dev/dri /dev/dri/*
# crw-rw---- 1 0  44 226, 0 Sep 16 16:55 /dev/dri/card0   <- 44 is the group id

docker run -d --device /dev/kfd --device /dev/dri \
  --group-add 44 \
  -v ollama:/root/.ollama -p 11434:11434 \
  --name ollama ollama/ollama:rocm

One AMD-specific trap worth knowing before you blame Docker: Ollama ships ROCm 7 libraries and needs a matching ROCm 7 kernel driver. On an older host driver, GPU discovery hangs for about 30 seconds, times out, and falls back to CPU — a failure that looks like a container permissions problem but is not. Ollama's docs point at amdgpu-install for the upgrade. Our AMD ROCm setup guide covers getting the host side right first, which is the order that saves time.

What this page does not claim

Three honest limits, so you know where to stop trusting us.

• We have not reproduced every branch on every distro. The commands here are transcribed from current vendor documentation and checked for existence — the CUDA image tag, the toolkit version, the Compose schema. Where a fix depends on hardware we do not have in front of us (a specific AMD card, a specific Windows driver), we tell you what to check rather than what result to expect.
• The ordering is a heuristic. We put "runtime not registered" first because it is the most common cause in the support threads we read, not because any distribution was ever counted.
• Snap-packaged Docker is its own world. On Ubuntu, a Docker installed from Snap has confinement rules that can block device access even after the toolkit is configured correctly. If snap list docker returns a row, that is worth eliminating before you keep editing daemon.json — reinstalling from Docker's own apt repository puts you back on the documented path.

Once the GPU is visible

If the canary prints your GPU table and your app still runs on CPU, the problem has moved into the application container.

ComfyUI, vLLM and Ollama each have their own device-selection logic that runs after Docker hands the GPU over — a wrong base image, a CPU-only PyTorch wheel, or a model that simply does not fit. Start with the ComfyUI guide for the image stack, Ollama setup mistakes for the configuration ones, and Docker Model Runner if you would rather let Docker manage models instead of maintaining your own Compose file.

Frequently Asked Questions

Sources

• NVIDIA Container Toolkit — installation guide (repository setup, nvidia-ctk runtime configure, rootless mode, sample workload)
• Docker Docs — "Enable GPU access with Docker Compose" and "GPU support in Docker Desktop"
• Ollama troubleshooting documentation (cgroup driver fix, AMD /dev/kfd group IDs, ROCm driver mismatch)
• Docker Hub — nvidia/cuda tag listing (check a base-image tag exists before you build against it)

Every command on this page is transcribed from the vendor documentation linked above. Nothing here is presented as a measurement of our own hardware.