Running LLMs on CPU Only: What Actually Works Without a GPU

Short answer: yes, you can run an LLM with no GPU at all — if you pick the right size. On an ordinary Intel i5 laptop, models under 2B generate 18-36 tokens/sec and 3-4B models around 7-10, while 7-8B models crawl at 3-4 (itsfoss, May 2026, 8-model CPU-only test). Start with ollama run qwen3:4b — a 2.5GB download — and know that your RAM bandwidth, not your core count, decides how fast it goes.

Nobody selling hardware wants to lead with this, but a huge share of local AI actually happens on machines with no dedicated graphics card: office laptops, home servers, that desktop with integrated graphics. This page is the honest map of what works, with every speed number sourced, the exact commands, and a clear line where CPU-only stops making sense.

---

The Short Answer

CPU-only inference is genuinely usable up to about 4B parameters, tolerable at 7-8B if you have fast DDR5, and a bad time beyond that — with one big exception (MoE models, covered below).

Three facts frame everything else on this page:

1. The software fully supports it. llama.cpp — the engine under Ollama, LM Studio, and most local tools — is "a plain C/C++ implementation without any dependencies" with "AVX, AVX2, AVX512 and AMX support for x86 architectures" per its README. CPU is not a degraded fallback mode; it is a first-class backend.
2. Speed is a memory problem, not a compute problem. For every token generated, the CPU streams essentially the whole model through RAM. InsiderLLM's CPU guide puts it plainly: "token generation speed scales almost linearly with memory bandwidth." Cores stop mattering surprisingly early.
3. Model size is the only lever that really moves the needle. Halving parameters roughly doubles speed. That is why the picks below cluster at 1-4B.

If you are still choosing hardware rather than working with what you have, start at our hardware hub instead — this page assumes the no-GPU machine is the machine.

---

What to Run: The Picks

Default pick: qwen3:4b (2.5GB). Speed pick: gemma3:1b (815MB). Reasoning pick: phi4-mini (2.5GB).

Tags and download sizes below verified against the Ollama library (August 2026):

Two notes on quantization. These are all standard Q4-class builds — the sweet spot for CPU, where smaller weights mean less data streamed per token (the full story is in our quantization guide). Gemma 3 additionally ships quantization-aware-trained variants (gemma3:1b-it-qat, gemma3:4b-it-qat) that cut memory further with less quality loss than post-hoc quantizing.

For what each model can actually do, the model pages have the detail — Phi-4 Mini is the most relevant one for CPU-only machines, since it is the largest of these picks that still fits the "reasoning without a GPU" brief.

---

Why RAM Bandwidth Decides Everything

A 7B model at Q4 is roughly 4.5GB of weights, and generating one token means reading essentially all of them. At 38 GB/s (typical DDR4 laptop) that caps you near 8 tok/s; at 77 GB/s (DDR5 desktop) near 14. No CPU upgrade fixes this — only faster memory or a smaller model does.

InsiderLLM published the cleanest version of this table (their measurements, 7B Q4 models):

Source: InsiderLLM, "CPU-Only LLMs: What Actually Works"

You do not have to take that table on faith — two lines of arithmetic reproduce it.

model size (GB) at Q4_K_M   ~=  0.6 x parameters in billions
throughput ceiling (tok/s)   =  memory bandwidth (GB/s) / model size (GB)

A 7B at Q4_K_M is therefore about 0.6 x 7 = 4.2 GB. On a 38 GB/s DDR4 laptop the ceiling is 38 / 4.2 = roughly 9 tok/s; on a 77 GB/s DDR5 desktop it is about 18. Call these what they are: arithmetic upper bounds. They assume the whole model streams exactly once per token and that nothing else is competing for memory, neither of which is true — so real output lands meaningfully below the ceiling, typically somewhere in the 50-80% band. The published figures above sit right there, which is the sanity check.

Run those two lines for your own RAM speed and your candidate model before you download 19GB of weights. You will know within a factor of two whether the thing is going to be usable.

Three practical consequences:

• Dual-channel RAM is free speed. One 16GB stick gives you half the bandwidth of 2x8GB. If your no-GPU desktop has a single stick, adding a second is the cheapest upgrade in local AI.
• DDR5 is a bigger deal for CPU inference than for gaming. The laptop-to-desktop jump in that table is nearly 2x tokens per second, from memory alone.
• More cores plateau fast. Once memory is saturated — often by 4-8 threads — extra cores just wait. This is why a modest Ryzen with fast RAM beats an old many-core Xeon workstation with slow RAM on small models.

The same math explains the exotic end: InsiderLLM's dual Xeon E5-2699 v4 build (128GB across 8 channels, ~$1,510-2,080 used) runs Llama 3.1 70B at ~3-5 tok/s — a 70B model with no GPU, at proof-of-concept speed. Interesting; not a daily driver.

If your speeds come in far below these bands, the bottleneck is usually something fixable — our local LLM slow fix guide walks the checklist.

---

Real Numbers: 8 Models, No GPU

On an ordinary Intel i5 laptop with 12GB RAM, everything up to ~2B is comfortable, 3-4B is workable, and 7-8B drops to "watch each word arrive."

The most complete recent public test is itsfoss's May 2026 run — 8 models, Q4_K_M quantization, CPU only, on exactly the kind of unremarkable laptop most people actually own:

Source: itsfoss, "Can You Run LLMs Locally Without a GPU? I Tested 8 Models on Linux", May 2026. Test machine: Intel i5-class laptop, ~12GB RAM.

Their own thresholds are the useful calibration: 3-5 tok/s feels "painfully slow," while 15-30 tok/s is "responsive enough for everyday use." Their verdict — 1-2B models "consistently offer the best balance" on CPU — is the right default to start from.

Calibrate for your machine: those are DDR4-laptop numbers. On a DDR5 desktop, InsiderLLM measured ~35 tok/s for a 3B model and ~14 for a 7B — roughly double. So a current desktop makes 7-8B models cross from painful into tolerable, and makes 3-4B genuinely pleasant.

One outlier worth knowing about: Microsoft's BitNet b1.58 (2B, 1.58-bit) hit ~34 tok/s on a laptop i7 using just 0.4GB of memory in InsiderLLM's testing — a preview of how far CPU-first architectures can go, though the ecosystem around it is still thin.

---

Setup: Ollama on CPU

There is no special CPU mode. Install Ollama, run a model, and it uses whatever you have — with no GPU detected, it simply runs on CPU.

Install (Linux; macOS and Windows installers are on the same download page):

curl -fsSL https://ollama.com/install.sh | sh

Run the default pick:

ollama run qwen3:4b

That is genuinely the whole setup. Two CPU-specific things worth knowing:

1. Verify what is actually running the model. Machines with an iGPU sometimes surprise you:

ollama ps

The PROCESSOR column shows 100% CPU, 100% GPU, or a split — straight from Ollama's FAQ.

2. Forcing CPU deliberately. For benchmarking, or when a weak iGPU is slower than your CPU, Ollama's GPU docs say to set the visibility variable to an invalid ID:

# NVIDIA
CUDA_VISIBLE_DEVICES=-1 ollama serve
# AMD
ROCR_VISIBLE_DEVICES=-1 ollama serve
# Vulkan devices off
OLLAMA_VULKAN=0 ollama serve

On tight-RAM machines, also mind context length: Ollama's FAQ notes OLLAMA_CONTEXT_LENGTH (and parallel request count) directly drive memory use. A 4B model that fits fine at a modest context can swap — and crater to unusable — if you crank the window on an 8-12GB machine. Full spec guidance lives in our Ollama system requirements guide.

---

Setup: llama.cpp

Use llama.cpp directly when you want the last 10-20% of CPU performance or fine-grained control; use Ollama when you want to get on with your day.

llama.cpp needs no GPU toolkit of any kind on CPU. Grab a pre-built binary from the releases page (or build from source per their build guide), then the current README quick start is:

# interactive chat, pulls the model from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# OpenAI-compatible local server with web UI
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Swap in any GGUF repo from Hugging Face — the -hf flag downloads and caches it. The prebuilt x86 binaries carry the AVX2/AVX-512 paths, which is where modern CPUs earn their keep on prompt processing.

Why bother over Ollama? Direct control over quantization choice per-file, first access to new architectures, and llama serve gives you an OpenAI-compatible endpoint with zero extra layers. For most people the honest answer is: Ollama, and revisit llama.cpp when you hit a wall.

---

The MoE Loophole

Mixture-of-Experts models break the size-speed rule: Qwen3-30B-A3B has 30B parameters but activates only ~3B per token — and a published CPU-only run on a Ryzen 9 9950X with DDR5-6400 (~88 GB/s) generated at over 30 tok/s.

That result comes from a benchmark discussion on the ubergarm/Qwen3-30B-A3B-GGUF Hugging Face repo — 30+ tok/s generation and 500+ tok/s prompt processing, CPU only. The trick: bandwidth limits you by active parameters per token, so an MoE streams like a ~3B model while reasoning closer to its full size (Qwen's own claim is that it outcompetes QwQ-32B with 10x fewer activated parameters).

The catches, honestly:

• You still need the RAM for all of it. qwen3:30b is a 19GB download; plan for 32GB+ of system memory.
• That 30 tok/s figure is a best-case config — current 16-core desktop, dual-channel DDR5-6400, tuned quants. A DDR4 laptop will not see it.
• Prompt processing on long documents is still CPU-bound pain compared to any real GPU.

But if your no-GPU machine is a recent desktop with 32-48GB of fast RAM, this is the single best quality-per-token-per-second deal in CPU land, and most people have never heard of it. Check your memory situation against our RAM requirements guide before pulling 19GB.

---

Honest Limitations

CPU-only is real, but it is not free lunch. Four things do not get better no matter how you tune:

1. Prompt processing is the hidden tax. Generation speed gets all the attention, but ingesting a long prompt (a big document, a full code file) is compute-heavy, and CPUs are far slower at it than any GPU. Chat feels fine; "summarize this 40-page PDF" means a long stare at a blinking cursor. RAG and long-context work are where CPU-only hurts most.
2. Dense 13B+ models are effectively out. At ~2 tok/s or less on consumer RAM, they are demos, not tools. The MoE loophole is the only practical route to big-model quality without a GPU.
3. Laptops throttle. Sustained inference pins every core; after a few minutes many thin laptops drop clocks and your 7 tok/s quietly becomes 5. Desktops and mini PCs hold their numbers, which is one reason headless boxes are so popular for local AI.
4. Battery life evaporates. CPU inference on battery is a fast way to a warm lap and a dead laptop. Treat it as a plugged-in activity.

None of this argues against starting on CPU. It argues for matching expectations: chat, drafting, quick questions, light coding help — great. Heavy document work and agent loops — you will feel the ceiling.

---

When a Cheap GPU Beats Your CPU

The moment you settle into daily use of a 7-8B model, roughly $300 of GPU makes it 5-10x faster. An 8GB card moves a 7-8B Q4 model entirely into VRAM at several hundred GB/s of bandwidth — the same bandwidth math from earlier, now working for you instead of against you.

The honest 2026 pricing picture: Intel's Arc B580 launched at $249 MSRP, but the memory shortage has street prices at roughly $309-320 as of mid-2026 (per bestvaluegpu.com and pc4games tracking) — still the cheapest current-gen path to 12GB of VRAM. Why everything costs more than MSRP right now is its own story: our GPU prices and the memory shortage breakdown covers it.

The decision rule we give readers:

• Stay CPU-only if 1-4B models cover your use, you run occasional queries rather than all-day sessions, or the machine simply cannot take a GPU (laptop, SFF, server).
• Buy the cheap GPU the week you notice yourself waiting on the model. What fits in 8GB is a solved question — our best LLMs for 8GB VRAM picks map it model by model.
• Skip the middle ground of paying flagship-GPU prices to run small models faster; that money buys more as RAM (for the MoE route) or a used 24GB card later.

CPU-only is the right on-ramp: it costs nothing, teaches you the whole stack, and tells you precisely — in tokens per second — what a GPU is worth to you.

---

Sources

• itsfoss — "Can You Run LLMs Locally Without a GPU? I Tested 8 Models on Linux" (May 2026) — the 8-model CPU-only speed table
• InsiderLLM — "CPU-Only LLMs: What Actually Works" — bandwidth-vs-speed measurements, dual-Xeon 70B build, BitNet figure
• llama.cpp repository — CPU backend support, quick-start commands
• Ollama docs — GPU-disable variables, ollama ps, context-length memory notes; model tags and sizes from the Ollama library
• ubergarm/Qwen3-30B-A3B-GGUF discussion — CPU-only MoE benchmark

---

FAQ