Why Is My Local LLM So Slow? 12 Fixes Ranked

Published April 11, 2026 • Updated August 23, 2026 • 17 min read

Short answer: single-digit tokens per second is almost always a memory problem, not a hardware problem. Either the weights did not fit in VRAM and part of every token is being computed on your CPU, or the GPU is not being used at all. One command settles it — ollama ps prints a PROCESSOR column that says exactly how the loaded model was split, for example 100% GPU or 43%/57% CPU/GPU. Anything other than 100% GPU means you are reading weights over the PCIe bus and system RAM instead of VRAM, and that is where your speed went.

Slow to download, not slow to generate? If ollama pull crawls, stalls at a percentage, or dies with max retries exceeded, that is a registry and network problem with a completely different cause and a completely different fix — nothing on this page will help. Read why ollama pull gets stuck or downloads slowly instead. Everything below assumes the model is already on disk and the generation is slow.

These 12 fixes are ordered as a diagnostic path, not as a statistic: the checks that take seconds and that explain the largest speed losses come first, and the rare, hardware-specific ones come last. Work down the list and stop at the first one that applies to you. Fixes 1, 2 and 5 are all variations on the same underlying problem — not enough free VRAM — which is why they sit at the top.

---

Fix 1: Model Too Large for VRAM (Partial CPU Offloading) {#fix-1}

Where it sits: check this before anything else.

This is the classic cause of slow local AI. Your model does not fully fit in GPU VRAM, so Ollama splits it: some layers run on the GPU, the rest run on the CPU. The CPU layers read their weights from system RAM, which on a typical desktop delivers roughly 90 GB/s against a mid-range GPU's 360 GB/s. Every token has to pass through both halves, so the slow half sets the pace.

Diagnose it:

# The direct answer: how was the loaded model actually split?
ollama ps
# NAME            ID          SIZE     PROCESSOR          UNTIL
# llama3.1:8b     ...         6.2 GB   38%/62% CPU/GPU    4 minutes from now
#                                      ^ anything but "100% GPU" means offloading

# Cross-check the card itself while text is generating
nvidia-smi
# "Memory-Usage" near the limit (e.g. 7800MiB / 8192MiB) = no room left

What you will see if this is your problem:

• ollama ps reports a CPU/GPU split rather than 100% GPU
• VRAM usage sits just under the physical limit
• The first token is slow too, because prompt evaluation is also split

Fix it — pick one:

# Option 1: use a smaller model
ollama pull llama3.2:3b
ollama pull phi4-mini        # 3.8B, leaves headroom on an 8GB card

# Option 2: use a smaller quantization of the same model
ollama pull llama3.1:8b-instruct-q3_K_M

# Option 3: shrink the context window, which frees VRAM for layers
OLLAMA_CONTEXT_LENGTH=2048 ollama serve
# or, inside an interactive session:
#   /set parameter num_ctx 2048

# Option 4: pin the layer count yourself (see Fix 6 for the Modelfile)

The math that decides it. At Q4_K_M a model costs roughly 0.6 GB per billion parameters, so an 8B model is about 8 × 0.6 = 4.8 GB of weights. On top of that sits the KV cache. For Llama 3.1 8B — 32 layers, 8 key/value heads, head dimension 128, keys and values, 2 bytes each in fp16 — that is 32 × 8 × 128 × 2 × 2 = 131,072 bytes per token, or 0.125 MB. A 4,096-token context therefore costs about 0.5 GB, and 32,768 tokens costs about 4 GB. Add roughly 0.5-1 GB of CUDA context and compute buffers and an 8 GB card has almost nothing spare at 4K context — while a 12 GB card is comfortable. The layer counts and head dimensions above come from the published model config; check any model's own config.json on Hugging Face if you want to redo this for a different architecture.

For model-by-model VRAM figures, see our VRAM requirements guide.

---

Fix 2: CPU Inference When a GPU Is Available {#fix-2}

Where it sits: second, and the single largest possible win.

Ollama is running entirely on CPU even though a perfectly good GPU is sitting idle. This happens when the driver is missing, the compute capability is unsupported, or the container/service cannot see the device.

Diagnose it:

# Step 1: does the OS see the GPU at all?
nvidia-smi
# If this errors, the driver is not installed or not loaded — nothing else matters

# Step 2: how did Ollama load the model?
ollama ps
# PROCESSOR showing "100% CPU" is the answer

# Step 3: the server log explains the decision
journalctl -u ollama --no-pager | tail -50     # Linux, systemd service
tail -50 ~/.ollama/logs/server.log             # macOS
# Look for the GPU discovery lines; "no compatible GPUs were discovered" is explicit

Note that OLLAMA_DEBUG=1 belongs to the server, not the client. On Linux, where Ollama usually runs as a systemd service, prefixing it to ollama run changes nothing — set it on the service, or read the log above.

Fix it:

# Linux: install/refresh the NVIDIA driver, then reboot
sudo apt update
sudo apt install nvidia-driver-550
sudo reboot

# Confirm the driver loaded
nvidia-smi

# If the driver is fine but Ollama still picks CPU, reinstall Ollama
curl -fsSL https://ollama.com/install.sh | sh

# macOS: Metal is automatic on Apple Silicon
system_profiler SPDisplaysDataType | grep Metal

Supported cards, minimum compute capability and the AMD/ROCm equivalents are listed in Ollama's own GPU support documentation — check your card there before assuming a driver problem.

Why it matters so much: generation is memory-bandwidth bound. Dual-channel DDR5-5600 gives 5,600 × 8 bytes × 2 channels = 89.6 GB/s. An RTX 3060 12GB is specified at 360 GB/s. That is a 4× gap in the exact resource that sets token speed, before any compute advantage is counted.

---

Fix 3: Quantization Chosen for the Wrong Reason {#fix-3}

Where it sits: third — cheap to check, easy to get backwards.

Smaller quantization does not automatically mean faster. Once a model already fits entirely in VRAM, dropping from Q4_K_M to Q2_K buys you a smaller file and worse output, and the throughput gain is far smaller than people expect because the kernels for the very low-bit formats are less well optimised than the 4-bit ones. The reason to quantize harder is to make the model fit — which is Fix 1 — not to make a model that already fits go faster.

Diagnose it:

# What quantization is this model actually using?
ollama show llama3.1:8b
# The Model block lists parameters, context length and quantization

# Or read it off the tag
ollama list

Fix it:

# Drop back to the standard Q4_K_M build
ollama rm llama3.1:8b-instruct-q2_K
ollama pull llama3.1:8b

Quantization reference for an 8B model. File size is bits-per-weight × parameters ÷ 8, which for 8 billion parameters makes the GB figure and the bits-per-weight figure land close together. The "fits in 8GB" column adds the 0.5 GB KV cache for a 4,096-token context calculated in Fix 1, plus roughly 0.8 GB of runtime buffers:

The k-quant formats and how they mix precision across tensors are documented in the llama.cpp k-quants pull request; sizes above are estimates from that arithmetic, so check the exact byte count on your model's tag page before planning around a tight fit. Our quantization explained guide walks through what each format does to the weights.

---

Fix 4: Thermal Throttling {#fix-4}

Where it sits: fourth — and much higher if you are on a laptop.

The signature is speed that starts fine and decays. Fast for the first half-minute, then progressively slower, then stable at a lower rate. The GPU is hot and has cut its own clocks to protect itself.

Diagnose it:

# Watch temperature, clock and power together while generating
watch -n 1 'nvidia-smi --query-gpu=temperature.gpu,clocks.gr,power.draw --format=csv,noheader'
# Falling clock while temperature climbs = throttling

# Ask the card for its own thresholds and the current throttle reason
nvidia-smi -q -d TEMPERATURE,PERFORMANCE
# Reports "GPU Slowdown Temp", "GPU Shutdown Temp", "GPU Max Operating Temp",
# and the active clock throttle reasons (SW/HW thermal slowdown, power cap)

# macOS Apple Silicon
sudo powermetrics --samplers thermal -i 2000 -n 10

Do not guess the threshold from a number you read online — every card reports its own, and nvidia-smi -q -d TEMPERATURE prints the exact slowdown temperature your GPU will act on.

Fix it:

# Cooling first, no commands required:
# - clear dust from the heatsink and fans
# - improve case airflow (pull the side panel off as a test)
# - laptop: a cooling pad, and lift the rear off the desk
# - confirm the fans actually spin up under load

# Cap power draw — sheds heat for a small throughput cost
sudo nvidia-smi -pl 120        # e.g. 120W on a 150W card

# More aggressive fan curve (Linux, X11, requires Coolbits)
nvidia-settings -a "[gpu:0]/GPUFanControlState=1" -a "[fan:0]/GPUTargetFanSpeed=80"

# Last resort: cap the clock
sudo nvidia-smi -lgc 300,1500

On desktops, dust and airflow usually explain it. On laptops the chassis is the limit and a cooling pad is the cheapest intervention. See our Ollama system requirements guide for what to plan around.

---

Fix 5: Background Processes Eating VRAM {#fix-5}

Where it sits: fifth — and it is Fix 1 wearing a disguise.

A browser with dozens of GPU-accelerated tabs, a game left running, or a desktop compositor all hold VRAM. Take enough of it and a model that used to fit no longer does, and Ollama quietly starts offloading layers — you get the Fix 1 symptom without changing anything about the model.

Diagnose it:

nvidia-smi
# The "Processes" table at the bottom lists every process holding VRAM
# and exactly how much each one holds — read your own numbers here,
# they vary enormously by tab count, resolution and desktop environment

Fix it:

# Close the biggest consumer in that table, or disable browser GPU acceleration:
#   chrome://settings → System → "Use graphics acceleration when available"

# Or end a specific GPU process by the PID nvidia-smi printed
kill -9 <PID>

# Then confirm the VRAM actually came back
nvidia-smi

The clean test: close everything except your terminal, run the model, and check ollama ps again. If PROCESSOR flips to 100% GPU and speed jumps, something else was holding the memory. Add applications back one at a time to find it.

---

Fix 6: Wrong Ollama GPU Layer Split {#fix-6}

Where it sits: sixth — for when the automatic split guesses badly.

Ollama decides how many layers to place on the GPU from the VRAM it believes is free. On multi-GPU systems, on cards shared with a display, or when another process grabs memory after the estimate, that guess can come out too conservative.

Diagnose it:

# The split Ollama chose
ollama ps

# The reasoning behind it, in the server log
journalctl -u ollama --no-pager | tail -80
# Look for the offload/layer lines, e.g. "offloading 20 repeating layers to GPU"

Fix it:

# Pin the layer count in a Modelfile
cat > Modelfile << 'EOF'
FROM llama3.1:8b
PARAMETER num_gpu 99
EOF
ollama create llama3.1-gpu -f Modelfile
ollama run llama3.1-gpu "test"

# If it fails to load or crashes with an out-of-memory error,
# lower num_gpu a few layers at a time until it loads.

You can set the same option per request through the API instead of building a model, by passing "options": {"num_gpu": 99} in the JSON body. There is no OLLAMA_NUM_GPU environment variable — num_gpu is a model parameter, so it goes in a Modelfile, in /set parameter num_gpu 99 inside an interactive session, or in the API options object.

The tradeoff: forcing every layer onto a card that cannot hold them turns a slow model into one that will not load. Reserve room for the KV cache using the per-token figure from Fix 1.

---

Fix 7: Models Stored on a Hard Drive {#fix-7}

Where it sits: seventh — obvious once you check, invisible until you do.

Storage speed does not change steady-state token generation once the model is resident in memory, but it dominates load time, and it dominates everything if the system starts swapping.

Diagnose it:

# Where does Ollama keep models?
ls -la ~/.ollama/models/

# Is that path rotational?
df -h ~/.ollama/models/
lsblk -d -o name,rota
# ROTA=1 is a spinning disk, ROTA=0 is solid state

# Is the machine swapping? If so, disk speed is now inference speed
free -h

The arithmetic: a 4.8 GB Q4_K_M 8B model at a 7200 rpm drive's typical sustained ~120 MB/s takes about 40 seconds to read. On a SATA SSD at ~500 MB/s that is about 10 seconds; on a PCIe 3.0 NVMe drive at ~3,000 MB/s it is under 2 seconds. Divide your own model size by your drive's sequential read rate to get your number.

Fix it:

# Move the model store to the fast disk
sudo systemctl stop ollama
mv ~/.ollama/models/ /path/to/ssd/ollama-models/
ln -s /path/to/ssd/ollama-models/ ~/.ollama/models
sudo systemctl start ollama

# Or point Ollama at it directly
export OLLAMA_MODELS=/path/to/ssd/ollama-models
# For the service, set it in /etc/systemd/system/ollama.service.d/override.conf

This matters most when you switch models often, when memory pressure pushes the system into swap, or when the boot drive is still a hard disk.

---

Fix 8: Context Window Larger Than You Need {#fix-8}

Where it sits: eighth — the fix that silently causes Fix 1.

The KV cache grows linearly with context length and lives beside the weights. Raising the context window is the easiest way to push layers off the GPU without touching the model at all.

Diagnose it:

# What context length is configured?
ollama show llama3.1:8b
ollama show llama3.1:8b --modelfile | grep num_ctx

# Watch memory grow as a conversation gets longer
watch -n 2 nvidia-smi

Using the per-token figure derived in Fix 1 — 0.125 MB per token for Llama 3.1 8B in fp16 — 4,096 tokens costs about 0.5 GB, 16,384 tokens about 2 GB, and 32,768 tokens about 4 GB. On an 8 GB card that last one alone is half your VRAM, and the layers it displaces move to the CPU.

Fix it:

# Server-wide default
OLLAMA_CONTEXT_LENGTH=4096 ollama serve

# Per model
cat > Modelfile << 'EOF'
FROM llama3.1:8b
PARAMETER num_ctx 4096
EOF
ollama create llama3.1-fast -f Modelfile

# Per session
#   /set parameter num_ctx 4096

Practical advice: size the context to the job. Interactive chat rarely exceeds a thousand tokens of history, and starting a fresh conversation releases the accumulated cache. Reserve the large windows for document work, and expect to pay for them in VRAM.

---

Fix 9: Power Management Holding the Clocks Down {#fix-9}

Where it sits: ninth — mostly laptops and mini PCs.

A power profile can cap clocks well below what the silicon can do. The symptom is uniform slowness with a cool GPU — no thermal decay, just a low ceiling.

Diagnose it:

# Current versus maximum graphics clock
nvidia-smi --query-gpu=clocks.gr,clocks.max.gr --format=csv

# Performance state: P0 is maximum, P8 is idle. During generation you want P0/P2.
nvidia-smi --query-gpu=pstate --format=csv

# Linux CPU governor — "powersave" hurts prompt processing
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Fix it:

# Linux: CPU to performance
sudo cpupower frequency-set -g performance

# Windows: Power Options → High performance (or Best performance in Settings)

# NVIDIA: enable persistence mode so the card does not drop state between runs
sudo nvidia-smi -pm 1

# If you want to raise application clocks, first ask what is supported:
nvidia-smi -q -d SUPPORTED_CLOCKS
# then set a pair from that list:
#   sudo nvidia-smi -ac <memory>,<graphics>

# macOS: System Settings → Battery → Low Power Mode: Off

---

Fix 10: Outdated GPU Drivers {#fix-10}

Where it sits: tenth — worth ruling out, rarely the answer.

Old drivers occasionally carry performance regressions or lack support for a newer card. This is far down the list because Ollama works across a wide range of driver versions, and a driver update is not a performance strategy.

Diagnose it:

nvidia-smi | head -3          # driver and CUDA version
apt list --upgradable 2>/dev/null | grep nvidia

Fix it:

sudo apt update
sudo apt install nvidia-driver-550
sudo reboot

# Or the graphics-drivers PPA for newer branches
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-560
sudo reboot

Set expectations honestly: do not assume a speed gain. Read the release notes for your branch and check whether they mention your card before you attribute anything to the update. If nvidia-smi works and Ollama reports 100% GPU in ollama ps, your driver is not your bottleneck.

---

Fix 11: WSL2 Memory Limits on Windows {#fix-11}

Where it sits: eleventh overall, but first if you run Ollama inside WSL2.

WSL2 does not hand the whole machine to Linux. It takes a fraction of system RAM by default, so a 32 GB machine can present a memory-starved environment to Ollama while Windows itself sits half empty.

Diagnose it:

# Inside WSL2
free -h
# If "total" is far below your physical RAM, WSL2 is the constraint

cat /mnt/c/Users/$USER/.wslconfig 2>/dev/null || echo "No .wslconfig found"

Fix it:

# In Windows PowerShell
notepad $env:USERPROFILE\.wslconfig

Add:

[wsl2]
memory=24GB
swap=8GB
processors=8

wsl --shutdown
# then reopen your WSL2 terminal

# Verify inside WSL2
free -h

The configurable keys and their defaults are documented in Microsoft's WSL configuration reference. A practical split is about three quarters of physical RAM to WSL2 and the rest to Windows. More context in our Ollama system requirements guide.

---

Fix 12: Docker Without GPU Passthrough {#fix-12}

Where it sits: last overall, but check it first if Ollama runs in a container.

A container gets no GPU access by default. Everything runs on CPU, and the symptom is identical to Fix 2 — except nvidia-smi on the host looks perfectly healthy, which is what makes it confusing.

Diagnose it:

# Can the container see the GPU?
docker exec -it ollama nvidia-smi
# "command not found" or "no devices were found" = no passthrough

docker inspect ollama | grep -i gpu

Fix it:

# 1. NVIDIA Container Toolkit
sudo apt install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 2. Recreate the container WITH the GPU flag
docker stop ollama && docker rm ollama
docker run -d --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# 3. Verify
docker exec -it ollama nvidia-smi

The load-bearing flag is --gpus all. Installation steps for the toolkit are in NVIDIA's container toolkit install guide, and the Docker specifics are covered in the Ollama FAQ.

---

The 60-Second Diagnostic {#diagnostic}

Run these in order. Stop at the first one that shows a problem.

# 1. How was the model split? (5 seconds)
ollama ps
# Anything other than "100% GPU" in PROCESSOR → Fix 1, or Fix 2 if it says 100% CPU

# 2. Is the card visible and how full is it? (5 seconds)
nvidia-smi
# Driver error → Fix 2. VRAM near the limit → Fix 1.

# 3. Who else is holding VRAM? (5 seconds)
# Read the "Processes" table in the output above → Fix 5

# 4. Is it throttling? (10 seconds)
nvidia-smi -q -d TEMPERATURE,PERFORMANCE
# An active thermal or power throttle reason → Fix 4

# 5. What quantization and context is loaded? (10 seconds)
ollama show llama3.1:8b
# Very low quant, or a context far larger than you need → Fix 3 / Fix 8

# 6. Are the clocks pinned low? (5 seconds)
nvidia-smi --query-gpu=pstate,clocks.gr,clocks.max.gr --format=csv,noheader
# Stuck in P5-P8 while generating → Fix 9

# 7. Running in Docker? (10 seconds)
docker exec -it ollama nvidia-smi 2>/dev/null || echo "No GPU in container"
# Fails → Fix 12

If none of that explains it, read the server log — it states which GPU library loaded and how many layers were offloaded:

# Linux
journalctl -u ollama --no-pager | tail -50

# macOS
tail -50 ~/.ollama/logs/server.log

# Windows (PowerShell)
Get-Content "$env:LOCALAPPDATA\Ollama\server.log" -Tail 50

---

What Speed Should You Expect? {#baseline}

There is a hard physical ceiling on single-stream generation, and it is easy to compute. Producing one token requires reading every weight once, so the fastest possible rate is memory bandwidth divided by model size. For a 4.8 GB Q4_K_M 8B model, using each vendor's published memory bandwidth:

Read this table correctly. These are ceilings, not measurements — nobody hits them, because attention, sampling, kernel efficiency and the KV cache all consume bandwidth too. Real single-stream output typically lands somewhere below the ceiling but in the same order of magnitude. That is what makes the table useful for troubleshooting: if you are at roughly half of your ceiling, your setup is behaving normally and further tuning is marginal. If you are at a tenth of it, something on this page is wrong — and it is almost certainly Fix 1 or Fix 2, because partial CPU offloading drags the effective bandwidth down to the DDR row.

The bandwidth figures are manufacturer specifications; confirm yours on the product page for your exact card, since board partners and laptop variants differ. For planning a build around these numbers, see our VRAM requirements guide and the RAM requirements guide.

---

Questions People Ask About Slow Local Models {#faq}

Why is my Ollama model generating only 2-5 tokens per second? Run ollama ps. If PROCESSOR shows a CPU/GPU split, the model did not fit in VRAM and part of every token is being computed against system RAM — roughly 90 GB/s instead of your card's several hundred. Use a smaller model, a smaller quantization, or a smaller context window (Fix 1). If it shows 100% CPU, the GPU is not being used at all, which is a driver or container problem (Fix 2 and Fix 12).

How many tokens per second should my GPU manage? Start from the ceiling: memory bandwidth ÷ model size. An RTX 3060 12GB at 360 GB/s running a 4.8 GB model cannot exceed about 75 tok/s, and in practice will land below that. Being under the ceiling is normal; being an order of magnitude under it is the signal to start at Fix 1.

How do I check whether Ollama is actually using my GPU? ollama ps is the direct answer — the PROCESSOR column reports the split for the loaded model. nvidia-smi is the cross-check: an Ollama process should appear in the Processes table holding VRAM, and GPU-Util should be non-zero while text is streaming. If neither is true, work through Fix 2.

Does quantization affect generation speed? It affects how many bytes must be read per token, so a smaller file can help — but only up to the point where the model already fits entirely in VRAM. Past that, the gain is modest and the quality cost is not. Quantize to fit, not to accelerate.

Why does my model start fast and then slow down? Two candidates. Thermal throttling, where the GPU cuts clocks as it heats — confirm with nvidia-smi -q -d TEMPERATURE,PERFORMANCE, which names the active throttle reason. Or a growing KV cache: at 0.125 MB per token for an 8B model, a long conversation steadily consumes VRAM until layers get pushed to the CPU. Starting a new conversation clears it.

How do I fix slow Ollama in Docker? Containers get no GPU by default. Install the NVIDIA Container Toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, restart Docker, then recreate the container with --gpus all. Verify with docker exec -it ollama nvidia-smi. Full commands in Fix 12.

Does WSL2 limit how much RAM Ollama can use on Windows? Yes — WSL2 allocates a fraction of physical RAM by default, so free -h inside WSL2 can show far less than the machine has. Set memory= in .wslconfig, run wsl --shutdown, and reopen the terminal. See Fix 11 and Microsoft's WSL configuration reference.

---

Conclusion

Slow local generation is nearly always one of three things: the model does not fit in VRAM (Fix 1), the GPU is not being used at all (Fix 2), or something else is holding the memory the model needed (Fix 5). All three are the same underlying problem, and ollama ps distinguishes them in about five seconds.

Run the 60-second diagnostic before changing anything. When speed drops suddenly on a setup that used to be fine, the cause is almost always something that changed around the model — a driver update, a new browser session, a larger context window — rather than hardware that degraded overnight.

For the wider set of failure modes, our local AI troubleshooting guide covers the errors that stop generation entirely, and the Ollama system requirements guide helps you plan the next upgrade.

---

Want to go deeper? Our courses include hands-on performance tuning labs where you learn to profile, diagnose, and optimize local AI inference on any hardware.