Raspberry Pi 5 LLM Guide: Which Models Fit and How Fast

Yes, a Raspberry Pi 5 runs real instruction-tuned LLMs — and its memory bandwidth tells you exactly how far that goes. The BCM2712 pairs with LPDDR4X-4267 on a 32-bit bus, which is 17.1 GB/s of theoretical peak bandwidth. Because generating one token requires reading every weight once, that single number caps every model on the board: roughly 21 tokens/sec for a 1B, roughly 8.6 for a 3B, roughly 3.5 for an 8B, before any overhead. The sweet spot is the 1B-3.8B tier on the 8GB board.

Below is the arithmetic for twelve models, the storage and cooling decisions that actually matter, and the exact setup commands. Every throughput figure here is a calculated ceiling from published specs — you can reproduce all of them with a calculator — not a stopwatch reading from someone else's board.

Quick Start: Ollama on Pi 5 in 6 Minutes

Fresh Raspberry Pi OS Bookworm 64-bit on a Pi 5. Active cooler attached. NVMe SSD strongly recommended; SD cards swap and die.

# 1. Update the OS and firmware
sudo apt update && sudo apt full-upgrade -y
sudo rpi-eeprom-update -a
sudo reboot

# 2. Install Ollama (official ARM64 build)
curl -fsSL https://ollama.com/install.sh | sh

# 3. Verify the install picked up your CPU cores
ollama --version
nproc  # Should print 4 (Cortex-A76 quad)

# 4. Pull and run the smallest credible chat model
ollama pull llama3.2:1b
ollama run --verbose llama3.2:1b "Greet me in two short sentences."

The --verbose flag makes Ollama print its own eval rate after the reply, so you get a real tokens-per-second figure for your board rather than trusting anyone's table. If it comes in under 2 t/s on a 1B, something is wrong: check that you booted from NVMe rather than SD, confirm free RAM with free -h, and make sure the active cooler is plugged into the fan header and spinning.

---

Why Run an LLM on a Pi at All?

The sceptical reaction is fair: a Pi 5 is two orders of magnitude slower than a desktop GPU. Three reasons it still makes sense.

1. Cost per always-on watt. Annual electricity is just average watts x 8760 / 1000 x your tariff. At a 7W average and $0.16/kWh that is 61 kWh, or about $9.80 a year — and a Pi genuinely sits in single-digit watts, where a desktop with a discrete GPU idles at 50W or more before it does anything. If you want a perpetually-on AI appliance in the house, nothing else is close.
2. True embedded deployment. A Pi fits in a project box on a shelf, survives a power cycle, and needs no supervision. You can hand it to a non-technical person and say "the box is the AI" and it will be true. No desktop AI machine does that.
3. Offline-first guarantees. With no Wi-Fi credentials and no Ethernet, a Pi is genuinely air-gapped — there is no firmware quietly phoning home. For privacy projects, journaling assistants and reference appliances, that is the whole point.

Pi 5 LLMs are not about throughput. They are about "good enough, always there, costs nothing to run."

Pi 5 8GB or 16GB: Which Should I Buy?

The 16GB board is the same BCM2712 SoC with double the LPDDR4X — same bandwidth, more capacity. For LLM use that distinction is the whole decision, because capacity governs what fits and bandwidth governs how fast it goes, and only one of them changes.

A note on price: DRAM contract prices rose sharply through 2026 on AI-server demand, and Raspberry Pi passed some of that through, so street prices sit above the launch figures in the table. Check raspberrypi.com for what it costs today rather than trusting any number in a blog post, including this one.

The decision itself is simple. Buy the 16GB if your use case includes RAG, long contexts, or several services on one board (Home Assistant plus Ollama plus an MQTT broker). Buy the 8GB if you want a chat assistant on sub-4B models — which, as the next section shows, is where the board is genuinely pleasant to use anyway.

The accessories are not optional:

• Active cooler. The official active cooler or the Argon One case. Raspberry Pi documents a soft thermal throttle at 80°C and a hard one at 85°C; a passively cooled board under continuous inference will reach them, and a throttled Pi loses clock speed exactly when you are waiting on it.
• 27W USB-C PD power supply. The Pi 5 asks for 5V/5A. Under-spec bricks cause brown-out reboots when CPU load spikes, which looks like random instability and wastes an evening.
• NVMe SSD via M.2 HAT. See below — the interface arithmetic makes this an easy call.

How Fast Can a Pi 5 Actually Generate Tokens?

Everything follows from one division:

tokens/sec ceiling = memory bandwidth (GB/s) / model file size (GB)

Pi 5 bandwidth = 4267 MT/s x 32 bits / 8 = 17.1 GB/s

To emit one token, the CPU reads every weight in the model exactly once. So the Pi's 17.1 GB/s is a hard wall no software trick gets past — and it is low, which is precisely why small models are the answer here. This is an upper bound, not a prediction: real output lands below it, because four Cortex-A76 cores also have to do the arithmetic that a GPU does in parallel, and on this board compute is often binding as well.

How to read this. Below 5 t/s an LLM feels slow; below 2 t/s only batch use cases make sense — overnight summarisation, cron jobs, anything where nobody is watching the cursor. The ceilings above put the whole 7B/8B tier in "batch, not chat" territory even before real-world overhead, which is the honest reason the 8GB board is enough for most people: the models the extra RAM unlocks are not models you want to sit in front of.

Two useful consequences. First, quantization moves the ceiling linearly — dropping Llama 3.1 8B from Q4_K_M (4.9GB) to Q3_K_S (3.7GB) shrinks the file about 24% and raises the ceiling by the same 24%, which is why Q3 is a rational choice on a bandwidth-starved board even though it costs quality. Second, any Pi benchmark you read claiming a 3B above roughly 8.6 t/s is claiming something the memory bus cannot do; treat it as a measurement error or a different quantization. The same reasoning applied to graphics cards is in GPU memory bandwidth for local LLMs.

To get your own real figure, add --verbose to any ollama run and read the eval rate it prints.

Do I Really Need an NVMe SSD?

Most "LLM on Pi" tutorials skip storage. The interface specs make the case on their own — model load time is just file size divided by the bus:

The Pi 5 exposes a single PCIe lane, so an NVMe drive on the HAT will not hit the numbers it would in a desktop — but a single Gen 2 lane is still an order of magnitude beyond an SD card, and the Gen 3 override roughly doubles it again. Raspberry Pi documents Gen 3 as outside the guaranteed envelope; it works on most boards and drives, and it is a one-line revert if yours is unhappy.

The other reason to leave SD behind is wear. If the kernel swaps — and an 8GB board running a 7B model will — SD cells die fast. Move swap to NVMe or disable it. Add roughly $25 for the HAT and $25 for a drive; this is the cheapest large improvement on the whole bill of materials.

Boot from NVMe, not SD:

# Set the bootloader to prefer NVMe
sudo rpi-eeprom-config --edit
# In the editor, ensure these lines are present:
#   BOOT_ORDER=0xf416     # NVMe -> SD -> USB -> repeat
#   PCIE_PROBE=1

# Optional: run the lane at PCIe Gen 3 (outside the guaranteed spec)
# Add to /boot/firmware/config.txt:
#   dtparam=pciex1_gen=3

# Save, exit, reboot
sudo reboot

# Verify NVMe is the root device
lsblk
# nvme0n1 should show /, /boot/firmware mounts

Is an Active Cooler Mandatory?

Effectively yes, and you do not need a benchmark to see why. Raspberry Pi documents the BCM2712's thermal behaviour directly: the SoC begins reducing clocks at 80°C and enforces a hard cap at 85°C. Sustained LLM inference pins all four Cortex-A76 cores at 100% indefinitely, which is the exact workload those limits exist for — unlike a compile or a video encode, it never lets up.

Once the board is throttling, it is running at reduced clocks for the whole session, so you pay the cost on every token. The official active cooler is roughly $5 and removes the question entirely. For a board that lives in a warm room — server closet, attic, a garage in summer — the Argon One style metal case is the better choice, because the chassis acts as a heat sink and the ambient headroom is what runs out first.

Watch your own board rather than guessing:

# Current SoC temperature
vcgencmd measure_temp

# Has it throttled? (0x0 means no)
vcgencmd get_throttled

# Log temperature every 10s during a long generation
while true; do echo "$(date +%T) $(vcgencmd measure_temp)"; sleep 10; done

How Do I Set Up Ollama on a Pi 5?

Two paths. Ollama for ease, llama.cpp for control. Most people end up wanting both.

Path 1: Ollama (recommended for most)

curl -fsSL https://ollama.com/install.sh | sh

# Tune for the Cortex-A76 quad core
sudo systemctl edit ollama
# Add under [Service]:
#   Environment="OLLAMA_NUM_THREADS=4"
#   Environment="OLLAMA_NUM_GPU=0"
#   Environment="OLLAMA_KEEP_ALIVE=24h"

sudo systemctl restart ollama

# Pull and run a model
ollama pull phi3.5:3.8b-mini-instruct-q4_K_M
ollama run --verbose phi3.5:3.8b-mini-instruct-q4_K_M

OLLAMA_KEEP_ALIVE=24h stops Ollama unloading the model from RAM after five idle minutes. Reloading a 2.4GB model over a single PCIe lane costs several seconds every time, and you do not want that on every query.

Path 2: llama.cpp from source

When you want finer control — custom KV cache size, context window, an OpenAI-compatible server — build it:

sudo apt install -y build-essential cmake git libcurl4-openssl-dev

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# ARM-optimized build with NEON SIMD enabled (default on aarch64)
cmake -B build -DLLAMA_CURL=ON
cmake --build build --config Release -j$(nproc)

# Download a model (already-quantized GGUF saves Pi time)
mkdir -p ~/models
cd ~/models
curl -L -o llama3.2-3b-q4.gguf \
  https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Run as an OpenAI-compatible API server
cd ~/llama.cpp
./build/bin/llama-server \
  -m ~/models/llama3.2-3b-q4.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -t 4 \
  -c 4096

-t 4 matches the Cortex-A76 core count. -c 4096 keeps context to 4K, which is the realistic ceiling on the 8GB board before the KV cache starts pushing you into swap. llama-bench, built alongside the server, will give you a clean tokens-per-second figure for any GGUF on your own hardware.

For production ARM tuning, the Arm Learning Path for llama.cpp documents the SVE and NEON flags — some apply to Cortex-A76, most to A78 and later.

What Can You Actually Build With It?

Project patterns that fit inside the bandwidth budget above:

1. Home Assistant voice assistant. Pi 5 16GB with Llama 3.2 3B, Wyoming, Whisper-tiny and Piper TTS. A wake word plus a short reply is a small number of tokens, which is exactly the workload the board is good at.
2. Offline note-taking helper. Pi 5 8GB with Phi-3.5 Mini and a terminal client. Dictate into Whisper, have the model clean up the transcript and suggest tags. Nothing leaves the device.
3. MQTT smart-home summariser. Qwen 2.5 1.5B on a cron job that reads the day's sensor events out of InfluxDB and writes a Markdown digest. Batch work, so throughput barely matters.
4. Air-gapped reference appliance. Llama 3.2 3B preloaded with first-aid, navigation and field-repair Q&A, running off a USB battery pack in a hard case. The offline AI survival kit covers the build.
5. Email triage backend. A Tailscale-tunnelled service that reads IMAP, scores priority with Phi-3.5 and writes labels back. Runs overnight, so 7 t/s is plenty.

The common thread: every one of these is either short replies or unattended batch work. Design around that and the Pi is a delight; ask it to write you an essay and you will watch a cursor.

For deeper recipes, see Local AI + Home Assistant and the private knowledge base guide.

Pi 5 vs Orange Pi 5 Plus vs Jetson Orin Nano

If you are choosing between single-board platforms specifically for LLM inference, compare memory — capacity for what fits, bandwidth for how fast:

The Jetson comparison is the honest one: NVIDIA publishes 102 GB/s for the Orin Nano Super, roughly six times the Pi 5's 17.1 GB/s, and it runs inference on a CUDA-capable GPU rather than four CPU cores. It is genuinely much faster, and it costs several Pi 5s. The Orange Pi 5 Plus has more RAM and a stronger CPU, but its GPU drivers still do not help with LLMs, so you are on CPU there too. The Pi wins on community, documentation and the certainty that a tutorial written last year still works.

Prices on all of these move constantly — check the vendor rather than any table.

What Goes Wrong and How Do I Fix It?

The failures that cost people evenings:

1. Boot loops from an under-spec power supply. The Pi 5 wants 5V/5A. Anything less browns out when four cores spike, and it presents as random instability rather than an obvious power fault. Use the official 27W brick or a PD supply that explicitly lists the Pi 5.

2. SD card corruption from swap. An 8GB board running a 7B model will swap, and SD cells wear out fast under it. Move swap to NVMe or turn it off:

sudo dphys-swapfile swapoff
sudo systemctl disable dphys-swapfile

3. Thread count defaults are wrong on ARM. Set OLLAMA_NUM_THREADS=4 explicitly. The auto-detect heuristic sometimes picks a higher count from logical CPUs that do not exist on a Cortex-A76 quad, and you lose cycles to lock contention.

4. Models pulled on x86 failing on ARM. Pull from the Pi itself with ollama pull. Some hand-copied GGUF files carry metadata that trips ARM loaders.

5. The Pi 4 active cooler does not fit a Pi 5. Different hole pattern. Buy the Pi 5 part by name.

6. Bookworm 64-bit only. 32-bit Raspberry Pi OS will not run Ollama. uname -m must print aarch64.

7. Do not raise the GPU memory split. The Pi 5 GPU is not used for LLM inference, so memory reserved for it is simply gone. Set it to the minimum:

sudo sed -i 's/^gpu_mem=.*/gpu_mem=8/' /boot/firmware/config.txt
sudo reboot

The official Raspberry Pi documentation covers the bootloader, NVMe and EEPROM commands referenced above. For Ollama-side failures, the Ollama troubleshooting guide covers the Linux causes that are not Pi-specific.

---

FAQ

Can a Raspberry Pi 5 run Llama 3?

The small variants, comfortably. Llama 3.2 1B and 3B both fit the 8GB board, with bandwidth ceilings of about 21 and 8.6 t/s respectively. Llama 3.1 8B at Q4 needs about 5.9GB resident, so it wants the 16GB board, and its ceiling is roughly 3.5 t/s — fine for batch summarisation, too slow to chat with. The 70B does not fit on either.

How many tokens per second does a Pi 5 generate?

Divide 17.1 GB/s by the GGUF file size. That gives roughly 25 t/s for TinyLlama 1.1B, 21 for Llama 3.2 1B, 8.6 for Llama 3.2 3B, 7.1 for Phi-3.5 Mini and 3.5 for Llama 3.1 8B — all upper bounds, with real output lower because the four CPU cores also have to do the arithmetic. For your own number, run ollama run --verbose and read the eval rate, or use llama-bench from a llama.cpp build.

Do I need an active cooler for LLM use?

Yes. Raspberry Pi documents clock reduction from 80°C and a hard cap at 85°C, and sustained inference is the workload most likely to reach them because it never lets up. The official cooler is about $5. Check with vcgencmd get_throttled — anything other than 0x0 means you are paying the tax.

Should I buy the 8GB or 16GB Raspberry Pi 5 for AI?

Buy 16GB if you want 7-8B models, RAG with a vector database, long context windows, or several services on one board. Buy 8GB if you want a chat assistant on sub-4B models — which is where the Pi is genuinely pleasant, since bandwidth caps the 7B tier at batch speeds regardless of how much RAM you have.

Is NVMe boot really required?

Practically, yes. The M.2 HAT gives a single PCIe lane at roughly 500 MB/s (or roughly 985 MB/s with the Gen 3 override), against roughly 25-100 MB/s for an SD card — so a 2GB model loads in a couple of seconds instead of tens. SD cards also wear out quickly once the kernel starts swapping. About $50 for HAT plus drive, and it is the best-value item in the build.

How much electricity does a Pi 5 LLM appliance use?

Work it out with average watts x 8760 / 1000 x your tariff. At a 7W average and $0.16/kWh that is roughly $9.80 a year. Measure your own average with a smart plug — it depends on your peripherals and duty cycle. The comparison that matters is the idle draw of the desktop you would otherwise leave on, covered in local AI power consumption.

Can I cluster multiple Pi 5s for bigger models?

Technically, via llama.cpp's RPC backend — but look at the link speed first. Gigabit Ethernet moves about 125 MB/s, against 17.1 GB/s of local memory, so the interconnect is well over a hundred times slower than the RAM it is standing in for. Splitting a model's layers across boards means every token crosses that gap. Clustering buys you capacity for a model that would not otherwise fit, not speed.

What about the Pi 4?

It works, slower. The Pi 4's LPDDR4-3200 on a 32-bit bus is about 12.8 GB/s against the Pi 5's 17.1, and the Cortex-A72 does less per clock than the A76, so both terms in the equation move against it. If you have one in a drawer, use it for a 1B model. If you are buying, buy the Pi 5.

Will the Pi 6 be much faster for LLMs?

Nothing is announced, and the platform has never targeted high-end inference throughput. Whenever a successor lands, the number to look for is memory bandwidth — that is what caps token generation, not the core count or any NPU headline. For a real step up today, the Jetson Orin family is the ladder: 102 GB/s and CUDA rather than 17.1 GB/s and four CPU cores.

---

Conclusion

The Raspberry Pi 5 is not the fastest local AI device. It is the most useful one in its class. For the price of the board plus a cooler and an NVMe drive you get a credible offline assistant that runs perpetually on a few watts. Phi-3.5 Mini and Llama 3.2 3B are capable enough for real work — smart-home commands, note summaries, journaling prompts, translation — and they sit in the part of the bandwidth budget where the board still feels responsive.

The mistake beginners make is treating the Pi 5 like a desktop. It is an appliance. Pick small models, respect the 17.1 GB/s ceiling, and design around what the board is actually good at: silence, low power, genuine air-gap potential, and an ecosystem that has supported these boards for over a decade.

If you have a Pi sitting in a drawer, today is a good day to plug it in. If you have outgrown it, the best mini PC for Ollama is the next rung, and models for 8GB machines covers the tier just above this one.