Local AI GPU Comparison: VRAM, Bandwidth & What Fits

Which GPU should I buy for local AI?

This is the whole guide in one table. It is sorted by VRAM (the thing that decides whether a model runs), then by memory bandwidth (the thing that decides how fast it runs). Specs are NVIDIA's published figures for each card; MSRP is the launch price, not what you will pay today.

How to read the last two columns. Dollars per GB is simply MSRP divided by VRAM — a $599 RTX 4070 with 12 GB is 599 / 12 = $50 per GB. "Largest model at Q4_K_M" assumes roughly 0.6 GB of weights per billion parameters at that quantisation, and reserves about 2 GB of the card for KV cache, context and the desktop. A 16 GB card therefore has ~14 GB for weights, and 14 / 0.6 is about 23 billion parameters.

Two corrections to the numbers you will see elsewhere, including in the older version of this page:

• The RTX 4080 SUPER has 16 GB, not 20 GB. There is no 20 GB consumer Ada card.
• A single 24 GB card does not run a 70B model at Q4. Llama 3.3 70B at Q4_K_M is roughly 43 GB of weights (Ollama lists the download at 43 GB). That is bigger than a 4090 and bigger than a 5090. You can force it with CPU offload or a 2-bit quant, but you are not getting a 24 GB card's speed out of it. If you want 70B at Q4 in VRAM you need ~48 GB, which in practice means two 24 GB cards — see the cheapest 70B build: dual RTX 3090 vs a single RTX 5090.

How much VRAM do I actually need?

Work backwards from the model, not from the budget. Pick the biggest model you want to run, find its Q4_K_M file size, add ~2 GB of headroom, and buy the cheapest card at or above that number.

The file sizes above come from the published download sizes in Ollama's model library, which are Q4_K_M by default, and they line up with the 0.6 GB-per-billion rule of thumb. For per-model maths including KV cache growth at long context, use the VRAM requirements reference or the interactive VRAM calculator.

Headroom is not optional. The KV cache grows with context length. At 8K context on a 7B–14B model you can budget 1–2 GB; at 32K on a 32B model it can exceed 6 GB. If the weights fill the card exactly, long conversations will spill into system RAM and throughput collapses.

Does memory bandwidth or CUDA cores decide tokens per second?

Bandwidth, almost entirely. During generation the GPU must read every weight in the model once per token produced, so single-stream decode speed is bounded by how fast the card can stream its own VRAM:

Ceiling (tokens/sec) = memory bandwidth (GB/s) ÷ model file size (GB)

That is an arithmetic upper bound you can compute yourself, not a benchmark. Real output lands below it — commonly 50–80% — because attention, KV cache reads, sampling and framework overhead all cost time the formula ignores. Use it to rank cards and to sanity-check a claim, never as a promised number.

Notice what the bandwidth column does to the value story: the RTX 4060 Ti 16GB is one of the cheapest ways to buy 16 GB, but at 288 GB/s it is the slowest card in this entire table — slower than a 12 GB RTX 3060. It buys capacity, not speed. That trade-off is the single most misunderstood thing in local-AI GPU shopping, and it is worked through in detail in GPU memory bandwidth and local LLM speed.

Which GPU is cheapest per GB of VRAM?

At MSRP, ranked cheapest first. Each figure is MSRP divided by VRAM.

The interesting result is the tie at the top of the stack: the RTX 5090, RTX 5080 and RTX 4080 SUPER all land on the same $62 per GB at MSRP, but the 5090 gives you that GB in one card with 1,792 GB/s behind it. Among the newer cards the RTX 5070 Ti is the standout — 16 GB and 896 GB/s for $47 per GB. Full workings and the used-market picture are in GPU price per GB of VRAM.

One large caveat: these are MSRPs, not street prices. Through 2026 the memory shortage has pushed actual retail well above MSRP on most of this list, and the gap is not uniform between cards — so re-run the division with the price you are actually being quoted. Current market conditions are tracked in why GPU prices are so high right now.

Is a used RTX 3090 still worth buying?

This is the most common question in this cluster, and the honest answer is that the 3090 is still the only affordable route to 24 GB.

What the specs say. The RTX 3090 has 24 GB and 936 GB/s. The RTX 4090 has 24 GB and 1,008 GB/s. That is a 7.7% bandwidth gap (1008 ÷ 936 = 1.077), and since decode is bandwidth-bound, single-stream generation speed between the two is far closer than the two-generation gap and the price difference suggest. The 4090 pulls ahead materially on prompt processing, image generation and fine-tuning, which are compute-bound rather than bandwidth-bound, and it is 100 W cheaper to run at full tilt (350 W vs 450 W TDP).

What that means in practice. If your workload is "chat with a 27B–32B model", a used 3090 gets you almost all of a 4090's token rate for a fraction of the outlay. If your workload is Stable Diffusion, long-document prompt processing or LoRA training, the 4090's extra compute is worth real money to you.

What to check before buying used. Ex-mining cards are common, VRAM temperatures on the 3090's back-side memory modules run hot by design, and remaining warranty is usually zero. The buying checklist is in the used GPU buying guide for AI, and a card-specific teardown is in the RTX 3090 for local AI guide. For the direct head-to-head, see RTX 4090 vs RTX 3090 for local AI.

We deliberately do not quote a used price here. Used 3090 listings move week to week and any figure printed on this page would be wrong within a month — check completed eBay listings on the day you buy, then divide by 24 to get your real dollars-per-GB.

Where does the RTX 50-series fit?

For local AI specifically — not gaming — the 50-series story is narrow.

The RTX 5090 is the only 50-series card that changes anything. It is the first consumer card past the 24 GB ceiling, with 32 GB of GDDR7 at 1,792 GB/s. That is 78% more bandwidth than a 4090 (1792 ÷ 1008 = 1.78) and 8 GB more capacity, which is the difference between a 32B model with a cramped context and a 32B model with room to breathe, or between Mixtral 8x7B fitting and not fitting.

The RTX 5080 and RTX 5070 Ti both ship with 16 GB. They are fast — 960 and 896 GB/s respectively, both above a 4080 SUPER — but they hit the same ~23B capacity wall as the 4070 Ti SUPER. Newer, faster 16 GB does not beat older, slower 24 GB when your model is 19 GB. If your target is 27B or 32B, neither card solves your problem at any price.

Official specifications for both generations are on NVIDIA's own pages: RTX 50 series and RTX 40 series. Every VRAM, bandwidth and TDP figure in the tables above is taken from those specification sheets.

Should I buy a GPU or a Mac / AMD unified-memory box?

For very large models, unified memory is now a legitimate alternative, because the capacity ceiling is the whole game.

• Apple Mac Studio (M3 Ultra) configures up to 512 GB of unified memory at ~819 GB/s. That capacity holds models no single consumer GPU can, at a fraction of the power draw of a multi-GPU rig. Trade-offs: no CUDA (you are on MLX or llama.cpp Metal), weaker prompt processing and time-to-first-token, and a steep price at the top configurations.
• AMD Strix Halo (Ryzen AI Max+ 395) offers up to 128 GB of unified memory at ~256 GB/s. Apply the roofline formula and you can see the consequence immediately: 256 GB/s against a 43 GB 70B model caps you around 6 tok/s before overhead. It is a capacity machine, not a speed machine — but it is a cheap, quiet, low-power way to hold a 70B model.
• Still want CUDA? A discrete NVIDIA card wins decisively on time-to-first-token, image generation and fine-tuning. Unified memory wins on capacity per watt.

The comparison across all three vendors is in AMD vs NVIDIA vs Intel for AI GPUs. On laptops, the calculus is different again — see laptop GPU VRAM for local AI.

Which GPU matches my workload?

Not sure which row is you? The interactive which GPU should I buy tool walks the same logic with your model list and budget, and the best local models for a 24GB GPU shows what a 3090 or 4090 can actually load.

What PSU and cooling do these cards need?

Inference is not gaming. A game loads the GPU in bursts; a long generation run holds it near its power limit for as long as the job lasts, so the PSU and the case have to survive a sustained draw rather than a spike.

• Size the PSU from TDP, with headroom. The rule of thumb is GPU TDP plus roughly 300 W for a modern CPU, board, drives and fans, then add 30% margin so the unit is not running at its efficiency cliff. A 450 W RTX 4090 lands around 1,000 W on that maths; a 575 W RTX 5090 lands closer to 1,150 W. NVIDIA's own recommendation for each card is on the specification pages linked above — use it as the floor, not the target.
• 12VHPWR / 12V-2x6 seating matters. The connector must be fully seated. Partial insertion under a sustained high-current load is the failure mode behind the melted-connector reports on high-TDP cards.
• Watch VRAM temperature, not just core temperature. GDDR6X modules throttle before the core does on sustained loads, and the RTX 3090's back-side memory placement makes it the worst offender.
• Prefer open-air dual or triple axial coolers over blower cards in anything but a rack chassis; blowers were designed for short bursts in dense servers, not hours of continuous load in a desktop.
• Capping the power limit is nearly free. Because decode is bandwidth-bound rather than compute-bound, reducing the board power limit costs proportionally less throughput than the wattage it saves. Test at 90% and 80% on your own card with your own model before deciding.

Common questions

Is VRAM or are CUDA cores more important? VRAM, without exception. A 16 GB card with modest cores beats an 8 GB flagship the moment your model exceeds 8 GB, because the alternative to fitting is offloading layers to system RAM across PCIe — and that costs an order of magnitude more than any core-count difference gains you.

Can I split one model across two GPUs? Yes. llama.cpp and vLLM both split layers across cards, and two 24 GB cards give you the ~48 GB that 70B at Q4 needs. You do not need NVLink for layer-split inference; PCIe is enough because only the activations cross the bus, not the weights.

Do I need the newest generation? No. The generation number is not on the axis that matters. A 936 GB/s 24 GB card from 2020 outruns a 288 GB/s 16 GB card from 2023 on every LLM workload, and holds bigger models while doing it.

Is more VRAM always better than more bandwidth? Up to the point where your model fits, yes — capacity is binary, and a model that does not fit does not run. Past that point the ranking flips and bandwidth is what you are buying.

Next steps

1. Decide the largest model you want to run, and look up its Q4_K_M size in the VRAM requirements reference.
2. Add ~2 GB of headroom and pick the cheapest row in the comparison table at or above that number.
3. Check bandwidth on that row against the roofline formula so you know what speed to expect.
4. Divide the price you are actually quoted by the VRAM, and compare it to the price per GB rankings before you commit.
5. Plan the rest of the build — CPU, RAM, PSU, case airflow — with the local AI hardware guide.