GPU Memory Bandwidth Table for Local LLM Speed

Local LLM token generation is limited by memory bandwidth, not by compute — producing one token means streaming the model's weights out of memory once. The ceiling is therefore bandwidth divided by model size: an RTX 5090's 1,792 GB/s against a 4.9 GB 8B Q4_K_M model works out to ~366 tok/s, an M4 Pro's 273 GB/s to ~56, and a dual-channel DDR5-5600 desktop's 89.6 GB/s to ~18. Nobody reaches the ceiling. Public llama.cpp benchmark data shows base Apple chips landing at 77-84% of it and the widest-bus Ultra chips at only 36-45%, which is why a machine with twice the bandwidth is never twice as fast.

This page is a lookup table. Every bandwidth figure is either published by the manufacturer or computed from the published bus width and data rate with the arithmetic shown, so you can check it. Every tokens-per-second figure in the ceiling columns is a derived physics limit, not a measurement — the only measured numbers on this page come from a named public benchmark thread and are labelled as such.

Why Does Memory Bandwidth Decide Local LLM Speed?

Because of what a transformer does when it writes one token at a time.

To produce the next token, a dense model multiplies the current hidden state by every weight matrix in the network. At batch size 1 that hidden state is a single vector, so each weight is loaded from memory, used for exactly one multiply-accumulate, and thrown away. The arithmetic-to-memory ratio is close to 1:1 — the worst possible case for a GPU, which is built for the opposite.

The consequence: the GPU spends almost all of its time waiting on memory, and its TFLOPS number is nearly irrelevant to how fast text appears. An RTX 5090 has 3.5× the CUDA core count of an RTX 5070 (21,760 vs 6,144) but only 2.7× the bandwidth (1,792 vs 672 GB/s), and it is the second ratio that predicts the token rate.

This is also why the answer to "why is my GPU so slow" is so often "it isn't — you're reading VRAM at exactly the speed the spec sheet promised". The genuinely fixable causes (CPU spillover, a model that doesn't fit, a bad backend) are covered in why your local LLM is slow and how to fix it; this page is about the floor underneath all of them.

Two things bandwidth does not decide, which the sections at the end cover in detail: how fast a long prompt is read (that step is compute-bound), and how much total throughput a server pushes when it batches many users at once.

How Do You Calculate the Tokens Per Second Ceiling?

One division:

ceiling tok/s  =  memory bandwidth (GB/s)  ÷  bytes read per token (GB)

For a dense model at batch size 1, "bytes read per token" is approximately the size of the model file sitting in memory. That gives a clean shortcut, because published GGUF sizes scale almost perfectly linearly with parameter count at a fixed quantisation:

• llama3.1:8b (Q4_K_M) is listed at 4.9 GB in the Ollama library → 4.9 ÷ 8 = 0.61 GB per billion parameters
• llama3.1:70b (Q4_K_M) is listed at 43 GB → 43 ÷ 70 = 0.61 GB per billion parameters

So Q4_K_M ≈ 0.6 GB per billion parameters, and a 32B at Q4_K_M is ~19 GB, a 14B ~8.5 GB, a 3B ~1.8 GB. Heavier formats scale from there: on TheBloke's Llama-2-7B card the same model is 4.08 GB at Q4_K_M and 7.16 GB at Q8_0, a 1.8× multiplier, with fp16 around 3.4×. The mechanics of each quant are in quantization explained.

Worked examples, so the arithmetic in the tables below is not a black box:

Read every ceiling number on this page as "physically impossible to exceed", not as "what you will get". Real single-stream output lands somewhere between roughly a third and four fifths of it, and the section on why is the most useful part of this page.

Bandwidth answers how fast. It says nothing about whether the model fits — that is a capacity question, and the Ollama model RAM and VRAM table and the VRAM calculator answer it. Check capacity first. A model that does not fit falls back to system RAM and the relevant bandwidth becomes the DDR figure in the fourth table, which is where 90% of "my GPU is slow" reports actually come from.

NVIDIA GeForce and RTX PRO Memory Bandwidth Table

Bandwidth for the RTX 50 series is taken directly from NVIDIA's own GeForce specification comparison. Earlier generations are computed from the published bus width and per-pin data rate: GB/s = data rate (Gbps) × bus width (bits) ÷ 8. The ceiling column is bandwidth ÷ 4.9 GB.

What this table is really telling you. Three pairs are worth staring at:

• RTX 5080 vs RTX 5070 Ti. Same 16GB, same 256-bit bus, and the 5080's faster 30 Gbps modules buy 7% more bandwidth. For token generation these are close to the same card. The differences that matter for local AI are elsewhere — see RTX 5090 vs 5080 for local AI.
• RTX 3090 vs RTX 5060 Ti 16GB. The five-year-old card has 2.1× the bandwidth and 8GB more VRAM. This is the entire argument for a used 3090, laid out in the RTX 3090 for local AI — the newer card wins on power draw, warranty and image generation, not on token rate.
• Tesla P40 vs RTX 4060 Ti 16GB. A GDDR5 card from 2016 out-streams a 2023 GDDR6 card by 20%, because a 384-bit bus at 7.2 Gbps beats a 128-bit bus at 18 Gbps. Bus width is doing more work than memory generation. What the P40 gives up is covered in the Tesla P40 for local LLMs.

Note the 128-bit tier generally. A 128-bit bus caps you near 450 GB/s no matter how fast the modules are, which is why an xx60-class card feels sluggish on a 14B even when the model fits comfortably.

AMD and Intel GPU Memory Bandwidth Table

Same method: computed as data rate × bus ÷ 8, except the HBM2 row where the bus is 4096 bits wide.

The row that surprises people is the MI50. A 2018 datacentre card with HBM2 has more raw bandwidth than any consumer card released since, because 4096 bits of bus at a lazy 2 Gbps beats 384 bits at 21 Gbps. Whether that translates into a usable machine is a software question, not a bandwidth question.

A caveat specific to AMD RDNA2 and RDNA3, and to NVIDIA's large L2: Infinity Cache and big on-die caches raise effective bandwidth for data that fits in them. A 4-19 GB weight stream does not fit in a 64-96 MB cache, and every weight is touched exactly once per token, so cache hit rates for token generation are close to zero. For this workload the DRAM figure is the honest one — which is exactly why the RX 7900 XTX at 960 GB/s outruns the newer RX 9070 XT at 640 GB/s, and why the Radeon AI PRO R9700 buys you capacity (32GB) rather than speed. Intel's position is the same story one tier down; see the Arc B580 for local AI.

Apple Silicon Memory Bandwidth by Chip

Apple publishes unified memory bandwidth in its own launch material, which makes this the best-documented family in the table. Figures marked (Apple) are quoted from Apple Newsroom; the rest are the values tabulated in the llama.cpp Apple Silicon benchmark thread linked in the next section.

Three things this table makes obvious:

1. M3 Pro went backwards. 150 GB/s against the M1 Pro and M2 Pro's 200 GB/s — Apple narrowed the bus on that generation. It is the one Apple Silicon chip that is slower at token generation than its two predecessors.
2. M5 is a base chip, not a Pro. 153 GB/s is a genuine ~28% lift over M4's 120, and Apple frames it that way, but it sits below an M1 Pro from 2021. Capacity caps at 32GB too. The detail is in the Apple M5 for local AI guide.
3. The tier gap is bigger than the generation gap. Moving Pro → Max roughly doubles bandwidth; moving M1 → M4 within the same tier barely changes it (M1 Max 400 → M4 Max 546 is +37% across three generations). If you are buying a Mac for local AI, the tier is the decision and the generation is a rounding error — the trade-offs are laid out in the Apple Silicon AI buying guide.

Unified Memory Boxes and Plain System RAM

The same division applies to anything that runs a model, including a machine with no discrete GPU at all. Bus widths here are per system, so the arithmetic is data rate × total bus width ÷ 8.

DGX Spark's 273 GB/s and 128GB of LPDDR5X are stated on NVIDIA's DGX Spark product page, alongside a 140W chip TDP. That is the single most important number about the machine and it explains the review consensus better than any benchmark: it is a capacity device, not a speed device. The full picture is in the DGX Spark review for local AI.

Strix Halo lands in exactly the same place. 256-bit LPDDR5X-8000 gives 8.0 × 256 ÷ 8 = 256 GB/s — roughly an M4 Pro, roughly a DGX Spark, roughly a quarter of an RTX 4090, with up to 128GB of capacity that no consumer GPU can touch. See the Ryzen AI Max+ 395 guide.

And the row that explains the most support requests: a normal desktop's dual-channel DDR5 is 5-10× slower than a mid-range GPU (89.6 GB/s against 448-960 GB/s), and 20× slower than an RTX 5090. The moment even one layer of your model spills out of VRAM, part of every token's weight read happens at ~90 GB/s instead of ~500 GB/s, and the token rate collapses far more than the small overflow would suggest. That is the mechanism behind almost every "it was fast yesterday" report. If you have no GPU at all and are working within these numbers deliberately, running LLMs on CPU only covers what is realistic.

Why Is My GPU Slower Than the Ceiling Says?

Because no memory system delivers 100% of its rated bandwidth to a real workload, and — this is the part that is rarely spelled out — the wider the bus, the smaller the fraction you get.

The best public dataset for this is the llama.cpp project's long-running "Performance of llama.cpp on Apple Silicon M-series" discussion (#4167), where contributors post llama-bench results for a 7B model in a consistent format. The table below takes the text-generation (TG, batch size 1) results for 7B Q4_0 from that thread, divides the same thread's bandwidth column by the published 7B Q4_0 file size of 3.83 GB (from TheBloke's Llama-2-7B-GGUF model card), and reports what fraction of the ceiling the measurement represents.

These are the only measured tok/s numbers on this page. They are other people's measurements on their own hardware, not ours.

Read down the "% of ceiling" column and the pattern is unmistakable:

• Base chips: 77-84%. A modest memory system is easy to saturate.
• Pro chips: 68-78%.
• Max chips: 52-72%.
• Ultra chips: 36-45%. An M1 Ultra has 11.8× the bandwidth of an M1 and produces 5.3× the tokens.

Why the efficiency falls off: getting a wide memory system to 100% utilisation needs enough parallel work in flight to keep every channel busy, and single-stream token generation is a thin, latency-sensitive, largely sequential workload. Ultra-class chips are also two dies stitched together, so some memory accesses cross the interconnect. Add the fixed per-token costs that do not shrink with bandwidth — attention over the KV cache, sampling, kernel launch overhead, the Python or Metal dispatch layer — and they dominate a bigger share of the budget as the memory time shrinks.

How to use this when reading the tables above. As a rough planning rule derived from the same data: multiply the ceiling by 0.8 for a small memory system, 0.7 for a mid one, 0.55 for a wide one, and 0.4 for the very widest. So the RTX 5090's ~366 tok/s ceiling on an 8B implies roughly 150-200 tok/s in practice, and the M4 Pro's ~56 implies high 30s. Those multipliers are extrapolated from Apple hardware; a discrete GPU with dedicated GDDR and a mature CUDA backend plausibly does better at the top end, but there is no comparably consistent public dataset for NVIDIA cards, so this page does not publish one.

What Can Actually Run a 70B, and How Fast?

A 70B at Q4_K_M is ~43 GB of weights, so this table is short. Capacity gates it first; bandwidth then sets the pace.

The multi-GPU row deserves its caveat spelled out. The common way to run a 70B on two 24GB cards is a layer split: the first half of the network lives on GPU 0, the second half on GPU 1, and each token passes through both in sequence. Only one card is reading weights at a time, so the effective bandwidth is one card's, not the sum. Tensor parallelism can genuinely use both memory systems at once, but it exchanges activations between cards on every layer, and over PCIe rather than NVLink that trade is often a wash. Two cards buy capacity reliably and speed only conditionally.

When Does Bandwidth Stop Predicting Speed?

Five cases where the division above gives the wrong answer. All five are real and all five come up constantly.

1. Mixture-of-experts models read only their active parameters. A sparse model routes each token through a small subset of experts, so bytes-read-per-token tracks the active parameter count, not the total. A 30B model with ~3B active parameters reads roughly 3 × 0.6 = 1.8 GB per token rather than ~18 GB — a ten-fold change in the only number that matters. All the experts must still fit in memory, which is why MoE is the format that makes 128GB unified-memory boxes interesting. Mechanics in mixture of experts explained.

2. Prompt processing is compute-bound, not bandwidth-bound. Reading your 8,000-token prompt happens in large batched matrix multiplies with high arithmetic intensity, so that phase is limited by TFLOPS. This is why an Apple chip can generate tokens at a respectable rate but take an uncomfortably long time before the first one appears, while a discrete GPU with far more compute chews through the same prompt in a moment. If your complaint is "it takes ages to start", bandwidth is not your problem.

3. Batching breaks the model entirely. Serve eight requests concurrently and one pass over the weights produces eight tokens. Aggregate throughput can be many times the single-stream ceiling, which is why server benchmarks from vLLM or SGLang look impossible next to this page's numbers. Per-user speed does not improve; total throughput does.

4. The KV cache is extra traffic that grows with context. The ceiling formula counts weights only. At long contexts the attention keys and values are also read every token, so effective bytes-per-token rises and speed decays as the conversation lengthens. Anyone who has watched a 32k-context chat get slower has seen this. Background in the KV cache and paged attention guide.

5. Speculative decoding gets more than one token per weight read. A small draft model proposes several tokens and the large model verifies them in a single pass, so accepted tokens come nearly free in bandwidth terms. On predictable text this legitimately beats the ceiling in this page's tables. See speculative decoding explained.

How Should You Use This Table When Buying?

In this order, because getting the order wrong is the expensive mistake:

1. Capacity first. A model that does not fit runs at DDR speed regardless of how good your GPU's bandwidth number is. Decide which model you want, size it at ~0.6 GB per billion parameters for Q4_K_M, add headroom for the KV cache, and only then look at the bandwidth column. Local AI VRAM requirements is the capacity-side companion to this page.
2. Bandwidth second, for how fast it will feel once it fits.
3. Compute third, and only if you also generate images, fine-tune, or routinely push long prompts. For chat-style token generation, TFLOPS is close to noise.
4. Software support as a veto. Bandwidth is worthless behind a backend that does not work on your card. The MI50's 1,024 GB/s is real; the ROCm support story around it is a separate and much longer conversation.

A blunt version: bandwidth per dollar and GB per dollar are the two numbers worth ranking cards by, and the current standings differ from the gaming standings enough to be worth checking against the best GPUs for local AI.

Honest Limitations

• The ceiling columns are arithmetic, not benchmarks. Bandwidth ÷ model size is an upper bound derived from published specs. No hardware was measured for those columns and none is implied.
• The only measured numbers are other people's, from llama.cpp discussion #4167, reproduced with attribution. That thread is community-contributed, so results vary by macOS version, llama.cpp build, thermal state and power mode — the ranges in the table reflect that spread rather than run-to-run noise on one machine.
• There is no equivalent dataset here for NVIDIA, AMD or Intel cards. A card-by-card measured tok/s table would need one of every GPU on a bench. Rather than publish numbers that cannot be sourced, this page publishes the physics ceiling and the efficiency multipliers derived from the Apple data, and says plainly that they are extrapolated.
• Bandwidth figures not marked as vendor-published are computed from bus width × data rate ÷ 8. Board partners ship factory-overclocked memory, and vendors occasionally revise module speeds mid-generation, so treat the last digit as approximate and check your specific card.
• The 0.6 GB per billion parameters shortcut is a Q4_K_M rule of thumb, calibrated against two published Ollama model sizes. Other quantisation formats, unusual architectures and models with large vocabularies or many shared parameters will deviate.
• Nothing here accounts for your backend. llama.cpp, Ollama, vLLM, MLX and TensorRT-LLM extract different fractions of the same hardware, and the gap between a good and a bad configuration is often larger than the gap between two adjacent GPUs.

FAQ

Is memory bandwidth or VRAM capacity more important for local AI?

Capacity, until the model fits — then bandwidth for everything after. They are not competing priorities so much as sequential gates. A 24GB card at 936 GB/s runs a 32B model well; a 12GB card at 1,792 GB/s cannot run it at all and will fall back to system RAM at roughly a tenth of the speed. Size the model first with the VRAM calculator, then compare bandwidth among the cards that clear the bar.

How many tokens per second will an RTX 5080 do?

Its physics ceiling on an 8B Q4_K_M model is 960 ÷ 4.9 = ~196 tok/s, and real single-stream output will be a fraction of that. This page does not publish a measured RTX 5080 figure because there is no public dataset for NVIDIA cards comparable to the llama.cpp Apple Silicon thread, and inventing one would be worse than leaving the gap. The ceiling is a genuine upper bound: whatever your setup produces, it will be below 196.

Why is my M1 Max slower than the bandwidth suggests?

Because wide memory systems are hard to saturate with single-stream token generation. In the llama.cpp #4167 results, M1 Max lands at 52-59% of its 400 GB/s ceiling while a base M1 reaches about 80% of its 68 GB/s. That is normal and not a misconfiguration. Ultra-class chips are worse again, at 36-45%.

Does a faster GPU core help if bandwidth is the same?

Barely, for token generation. Two cards with identical bandwidth and different shader counts will produce text at close to the same rate, because both spend most of their time waiting on memory. The extra compute does show up in prompt processing, image generation and fine-tuning — workloads with high arithmetic intensity where the same weights get reused many times.

Do two GPUs double my bandwidth?

Not with the usual layer split, which puts different layers on each card and runs them in sequence — only one card reads weights at a time, so effective bandwidth stays at one card's figure. Tensor parallelism can use both memory systems simultaneously but exchanges activations between cards on every layer, and over PCIe that overhead frequently cancels the gain. Two cards are a reliable way to buy capacity and an unreliable way to buy speed.

Why is Strix Halo or DGX Spark slower than my old gaming GPU?

Because 256 GB/s and 273 GB/s are roughly a quarter of an RTX 4090's 1,008 GB/s. Both machines exist to offer 128GB of memory in a small, low-power box, not to generate tokens quickly. They will run models your GPU cannot load at all, at a pace your GPU would beat on anything that fits in 24GB.

Does quantisation make the model faster, or just smaller?

Both, and for the same reason. Cutting the bytes per weight cuts the bytes read per token, which cuts the time spent waiting on memory. Going from Q8_0 to Q4_K_M shrinks the same 7B from 7.16 GB to 4.08 GB on TheBloke's model card, so it raises the ceiling by about 1.8×. That is why quantisation is the single most effective speed lever on bandwidth-limited hardware, and why the quality trade-off is worth understanding properly — see quantization explained.

Sources

• NVIDIA — GeForce graphics card specification comparison (RTX 50 series memory config, interface width and bandwidth)
• NVIDIA — DGX Spark product page (128GB LPDDR5X, 273 GB/s, 140W chip TDP)
• Apple Newsroom — M4 Pro and M4 Max (273 GB/s, up to 546 GB/s)
• Apple Newsroom — M4 family MacBook Pro (M4 at 120 GB/s)
• Apple Newsroom — M3 Ultra ("over 800GB/s", up to 512GB unified memory)
• Apple Newsroom — M5 (153 GB/s, 32GB max)
• Apple Newsroom — M1 Pro and M1 Max (200 GB/s and 400 GB/s)
• llama.cpp discussion #4167 — Performance of llama.cpp on Apple Silicon M-series (measured TG tok/s and the per-chip bandwidth column)
• TheBloke/Llama-2-7B-GGUF (Q4_0 = 3.83 GB, Q4_K_M = 4.08 GB, Q8_0 = 7.16 GB)
• Ollama library — llama3.1 tags (8B Q4_K_M = 4.9 GB, 70B Q4_K_M = 43 GB)