Dual GPU vs One Big GPU for Local LLMs: What Pools

Two 16GB cards give you 32GB for language model weights and 16GB for everything else. LLM runtimes — llama.cpp, Ollama and vLLM — will split a model across both cards; ComfyUI, video generation and the default multi-GPU fine-tuning path will not, because they either run on one device or keep a full copy of the model on each. So the decision is not "2x16GB or 1x32GB" in the abstract. It is: if your main workload is text, two cards buy you real capacity; if it is images, video or training, the second card mostly buys you a second queue.

This page is the tradeoff table, not the setup guide. If you have already decided and just need the flags and environment variables, go straight to our multi-GPU Ollama configuration walkthrough. If you specifically want a 70B and are choosing between two used 3090s and one 5090, the dual 3090 vs 5090 cost breakdown does that one build in detail. What follows is the part that comes before both: what pools, what does not, and what the second card costs you in lanes, watts and slots.

Every number below is either quoted from a vendor spec page or an official doc, or computed from one with the arithmetic shown. Where a claim would need hardware to verify, it is stated qualitatively and labelled as such.

What Actually Pools Across Two Cards and What Does Not?

This is the table that decides the purchase. "Pools" means the two cards' VRAM behaves like one larger pool for that workload. "No" means a 32GB model still will not run, no matter how many 16GB cards you own.

Read the middle of that table twice. The single most common expensive mistake is buying a second GPU for image or video generation. ComfyUI's argument parser has --cuda-device ("Set the ids of cuda devices this instance will use, as a comma-separated list (e.g. '0' or '0,1'), or 'all' to leave all currently visible devices available") and --default-device ("Set the id of the default device, all other devices will stay visible"). Neither splits a diffusion model. There is no equivalent of llama.cpp's layer split in the box.

The most popular community extension for this is honest about it. The ComfyUI-MultiGPU README states plainly: "This enhances memory management, not parallel processing. Workflow steps still execute sequentially, but with components (in full or in part) loaded across your specified devices." That is genuinely useful — moving the text encoder and VAE off your main card frees VRAM for the latent — but it is not a 32GB card. If image generation is your bottleneck, the best GPU for image generation comparison is the page you want, and the answer there is almost always one bigger card.

How Does Each Runtime Split a Model?

Three different mechanisms, three different consequences. The flag names and help text below are quoted from the projects' own documentation.

The word doing the work in the llama.cpp table is pipelined. The default layer mode does not run both cards at once on a single request — it hands the activations from the last layer on GPU 0 to the first layer on GPU 1 and waits. That is why the transfer over PCIe is small (one activation tensor per handoff, not whole weight matrices) and why it works acceptably on a slow link. It is also why it does not double your speed.

row and the experimental tensor modes are the parallel ones, and they are the ones that hammer the interconnect. vLLM's parallelism and scaling docs give the guidance that matters for a desktop build: "if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead." Consumer builds almost never have NVLink — see the mixed-hardware section below — so on a two-card desktop the low-communication mode is usually the right one, which is also the mode that does not multiply your throughput.

One vLLM constraint that catches people at 3 GPUs: the tensor parallel size has to divide the model's attention head count. The check lives in vllm/config/model.py and the error reads, verbatim, "Total number of attention heads (N) must be divisible by tensor parallel size (M)." A model with 32 or 64 heads is happy with 2, 4 or 8 cards and will refuse 3.

Do Two GPUs Make Tokens Come Out Faster?

For one user typing one prompt: usually not, and often the opposite. This is the claim most build threads get wrong in both directions, so here is what can actually be sourced rather than a scaling factor nobody has measured on your hardware.

Ollama's own FAQ describes its behaviour and, in the same breath, tells you which arrangement it considers fastest: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU. This typically provides the best performance as it reduces the amount of data transferring across the PCI bus during inference. If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs." Splitting is the fallback for models that do not fit, not an optimisation.

The arithmetic behind that is the bandwidth gap. Single-stream token generation is dominated by reading the weights out of VRAM once per token, so it tracks memory bandwidth. From NVIDIA's own spec comparison:

Two RTX 5080s hold the same 32GB as one RTX 5090. But under the default pipelined layer split, only one card is working on your token at a time, and that card reads its half of the weights at 960 GB/sec instead of 1792 GB/sec. You have bought capacity, not bandwidth. This is the single clearest argument for one bigger card when both options reach the same total VRAM.

Now the interconnect. A PCIe 5.0 lane runs at 32 GT/s with 128b/130b encoding, so 32 x (128/130) / 8 = 3.94 GB/s per lane: about 31.5 GB/s at x8 and 63 GB/s at x16. A PCIe 4.0 lane is half that, 1.97 GB/s, so a PCIe 4.0 x4 slot gives you 7.9 GB/s. Against the 5090's 1792 GB/s of VRAM bandwidth, that x4 link is 1792 / 7.9 = roughly 1/227th the speed. Even a full PCIe 5.0 x8 slot at 31.5 GB/s is about 1/57th.

That ratio explains the whole design space:

• Pipelined layer split survives it because it only moves one activation tensor per handoff.
• Tensor parallelism does not, which is exactly why vLLM's docs point you at pipeline parallelism without NVLink.
• Model loading is bound by it either way, so a card on an x4 chipset slot takes noticeably longer to swap models.

Where two cards genuinely do multiply throughput is concurrent requests — several people or several agents hitting the server at once, where each card can be working on a different batch. That is a different question from "does my chat go faster", and it is the question our Ollama multi-user rate limiting notes are aimed at.

What we are not going to tell you is a tokens-per-second multiplier for your combination. We do not own this hardware and there is no vendor figure for it. Anyone quoting you "1.8x with two cards" measured one model, one quantisation, one context length and one PCIe topology.

What Fits on 32GB, and What Needs 48GB?

This is where the decision usually gets settled, because model files are a hard boundary. The sizes below are the actual published file sizes on Hugging Face, converted from the listed bytes into GiB (the unit your GPU reports VRAM in — a "42.5 GB" file is 39.6 GiB of VRAM before any context).

Three things fall out of that table that no amount of forum argument changes:

1. The 32GB tier is a real tier, and 2x16GB reaches it. Everything in the 24B-32B class fits comfortably in 32GB with room left for context. If your target is Qwen3 32B, two 16GB cards and one 32GB card both work.

2. A 70B does not fit in 32GB, from either direction. Llama 3.3 70B at Q4_K_M is 39.6 GiB. Two 16GB cards give you 32 GiB; one RTX 5090 gives you 32 GB. Neither is enough. Even Q3_K_M at 31.9 GiB effectively fills a 32GB card before the KV cache, the CUDA context and your desktop compositor get a byte. If a 70B at a sane quantisation is the goal, 2x24GB is the entry point, and that is a genuine win for the dual-card build. That is the case the dual 3090 vs 5090 breakdown exists for, and it is the strongest argument for two cards on this entire page.

3. Above 48GB the consumer answer runs out. gpt-oss-120b at Q4_K_M is 58.5 GiB — three 24GB cards, or a unified-memory machine, or CPU offload with the speed penalty that implies. Our VRAM requirements reference and the Ollama model RAM and VRAM table list this per model if you want to check your specific target before committing to a build.

Note that the quantisation you pick moves the boundary more than the second GPU does. Dropping Llama 3.3 70B from Q4_K_M to Q3_K_M saves 8.25 GB of file size — a bigger swing than most upgrade decisions, and free.

Will Your Motherboard Give the Second Card Real Lanes?

Assume it will not until you have read your board's expansion slot table. This is the cost that never appears in a parts list, and mainstream boards differ enormously.

Two real examples from ASUS's own spec pages, both AMD X870E, both current, both marketed at enthusiasts:

Same chipset, same brand, same generation — and one of them puts your second GPU on a link roughly a quarter the width and half the generation of the first. Using the arithmetic from the previous section, that is 7.9 GB/s versus 31.5 GB/s.

Does an x4 second slot ruin it? For the default pipelined layer split, it is tolerable — the handoff is small. For tensor-parallel serving it is the wrong platform. For model loading it is simply slower, every time. What it definitely does is remove any future option to run the parallel split modes.

Three more physical checks before you buy, none of which are exotic and all of which have sunk builds:

• Slot pitch versus card thickness. A board that offers x8/x8 still has to physically place the slots far enough apart. Read the card's stated slot width in the vendor spec and count the slot positions on the board diagram. Triple-slot cards in adjacent x16 slots do not fit, and if they do, the top card's fans are pressed against the bottom card's backplate.
• What the second slot steals. On many boards, populating the second x16 slot disables one or more M.2 sockets or SATA ports, because those lanes are shared. That is in the same expansion-slot footnote you are already reading.
• Case depth and airflow. Two cards stacked means the top one inhales the bottom one's exhaust. This is the practical reason a lot of dual-GPU builds end up in an open frame, and why our homelab AI server build notes spend as much time on chassis as on silicon.

What Does the Second Card Cost in Watts?

Vendor TDP figures, plus NVIDIA's own minimum system power recommendation for a single card. The last column is arithmetic, clearly marked as such — NVIDIA publishes a figure for one card, so adding the second card's full TDP on top is the floor, not a recommendation.

TDP and minimum system power figures are from NVIDIA's 50 Series and 30 Series spec pages.

The two rows worth staring at:

2x RTX 5080 versus 1x RTX 5090. Identical 32GB. The dual build draws 720 W of GPU TDP against the 5090's 575 W — 145 W more — while each card reads memory at 960 GB/sec instead of 1792. It needs two slots, x8/x8 lanes, and a bigger power supply. For LLM inference specifically, this is the configuration that is hardest to justify. The RTX 5090 vs 5080 comparison goes into what each card does outside of LLM work.

2x RTX 5060 Ti 16GB. 32GB for 360 W total — less GPU power draw than a single 5090, and by a wide margin the cheapest route to 32GB. The catch is right there in the bandwidth table: 448 GB/sec per card. This build is about fitting a 32B model, not about generating quickly. That is a legitimate goal and a legitimate choice, as long as you make it with your eyes open.

Running costs compound with idle draw and hours-per-day, which is a different calculation entirely — our local AI power consumption breakdown works through it.

So Should You Buy 2x16GB or 1x32GB?

The decision by workload, with the reasoning already established above rather than a vibe.

That fine-tuning row deserves its caveat spelled out, because it is the most misunderstood line in this whole topic. Hugging Face Accelerate's FSDP guide maps the sharding strategies onto DeepSpeed's ZeRO stages: FULL_SHARD "Shards optimizer states, gradients and parameters" (ZeRO-3), while NO_SHARD maps to ZeRO Stage-0 — "No sharding wherein each GPU has full copy of model, optimizer states and gradients." Plain data-parallel training is the second one. Two 16GB cards running DDP give you a 16GB training budget and a bigger effective batch size, not a 32GB training budget. You have to opt into FSDP or DeepSpeed, configure it, and pay the communication cost over that PCIe link.

Even the tooling built to make this easy says it is not yet easy. Unsloth's multi-GPU documentation notes that it "currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed", that model splitting is available via device_map = "balanced", and, candidly, that "the process can be complex and requires manual setup" with simpler official support still to come. Budget the afternoon.

What About Mixed Cards, AMD, and Two Strix Halo Boxes?

Mismatched NVIDIA cards. llama.cpp gives you --tensor-split for exactly this: "fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1". A 24GB card paired with an 8GB card is a 3:1 split. It works, and the pipeline runs at the pace of the slower stage — pairing a fast card with a slow one gives you the capacity of both and something closer to the speed of the slow one. If you are shopping the used market to make a pair, the used GPU buying guide covers what to check.

NVLink is not coming back. NVIDIA's 30 Series spec page lists "NVIDIA NVLink (SLI-Ready)" as Yes for the RTX 3090 and 3090 Ti and a dash for everything below. It is not listed on any 50 Series card. This is a quiet but real point in favour of two used 3090s over two new mid-range cards for anyone who wants to run tensor-parallel serving: the 3090 pair is the last consumer configuration with a proper GPU-to-GPU link. Our RTX 3090 for local AI page covers the rest of that card's case.

AMD. Ollama documents ROCR_VISIBLE_DEVICES for limiting which Radeon GPUs it uses, mirroring CUDA_VISIBLE_DEVICES on NVIDIA. It also documents the mixed-architecture case explicitly: "If you have multiple GPUs with different GFX versions, append the numeric device number to the environment variable to set them individually. For example, HSA_OVERRIDE_GFX_VERSION_0=10.3.0 and HSA_OVERRIDE_GFX_VERSION_1=11.0.0." Two identical Radeon cards are the sane configuration; two different ones are a supported-but-fiddly one. Start from the AMD ROCm local LLM setup guide before you buy the second card, not after.

Two Strix Halo machines are not a dual-GPU build. A Ryzen AI Max+ APU box pools CPU and GPU access to one large unified memory space inside a single machine — that is its whole appeal, covered in the Strix Halo AI Max+ 395 guide. Two of them are two computers. Joining them means distributed inference over a network, not over PCIe. llama.cpp does ship a mechanism for this: the RPC backend "communicates with one or several instances of ggml-rpc-server and offloads computations to them". Read its own warning before you plan around it, though — the RPC backend README states it is "currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!" For managing several machines as a pool with something more production-shaped, GPUStack is the tool built for that job.

Honest Limitations

• No hardware was tested for this page. Every figure is quoted from a vendor spec page or an official doc, or computed from one with the arithmetic printed inline. Where the honest answer needed a benchmark we do not have, the page says so instead of inventing a number.
• There is no tokens-per-second multiplier here on purpose. Scaling depends on the model, quantisation, context length, split mode, PCIe topology and driver version. A single quoted multiplier would be wrong for almost every reader.
• Motherboard lane layouts change per model, not per chipset. The two ASUS boards above are illustrations of how wide the gap is, not a claim about X870E generally. Read your own board's expansion slot table.
• PSU numbers are TDP arithmetic, not a load measurement. Vendor TDP plus vendor minimum-system-power is a floor. Real builds add CPU, drives and transient behaviour on top.
• Flags drift. The llama.cpp, vLLM, Ollama and ComfyUI options quoted here were read from their current documentation. Check --help on your build before copying anything.
• The AMD side is thinner than the NVIDIA side, because AMD does not publish an equivalent minimum-system-power column and the Radeon spec page did not respond when this was written. Rather than fill that gap with estimates, it is left empty.

FAQ

Can Ollama use multiple GPUs?

Yes, automatically, with no configuration. Ollama's FAQ describes the logic: if the model fits entirely on one GPU it loads there, because that "typically provides the best performance as it reduces the amount of data transferring across the PCI bus during inference"; if it does not fit, "then it will be spread across all the available GPUs". You can restrict which cards it sees with CUDA_VISIBLE_DEVICES on NVIDIA or ROCR_VISIBLE_DEVICES on AMD.

Is 2x RTX 5080 better than 1x RTX 5090 for local LLMs?

For a single user, generally no. Both reach 32GB, but NVIDIA rates the 5080 at 960 GB/sec of memory bandwidth against the 5090's 1792 GB/sec, and the default layer split runs the cards one after another rather than together. The pair also draws 720 W of TDP against 575 W, needs an x8/x8 board and two slots. The dual build wins on price per gigabyte and on concurrent-request throughput, not on single-stream speed.

Do two GPUs double tokens per second?

No. llama.cpp's default --split-mode layer is documented as "pipelined" — it splits layers and KV cache across cards and runs them in sequence, so only one card works on a given token at a time. The parallel modes (row, and the experimental tensor) exist, but they need far more interconnect bandwidth, which is why vLLM's docs recommend pipeline parallelism over tensor parallelism when there is no NVLink. Two cards reliably multiply throughput for concurrent requests, not for one conversation.

Will a second GPU help with Stable Diffusion, FLUX or video generation?

Not for making one image bigger or faster. ComfyUI's --cuda-device flag selects which devices are visible; there is no option to split a diffusion model across cards. The ComfyUI-MultiGPU extension can move the VAE, CLIP and parts of the UNet to a second device to free VRAM on the first, but its own README says this "enhances memory management, not parallel processing" and that "workflow steps still execute sequentially". Two cards do let you run two independent generation queues.

Will 2x16GB let me fine-tune a model that needs 32GB?

Only if you configure sharding. Hugging Face Accelerate maps FSDP's FULL_SHARD to DeepSpeed ZeRO-3, which "Shards optimizer states, gradients and parameters", and NO_SHARD to ZeRO Stage-0 — "No sharding wherein each GPU has full copy of model, optimizer states and gradients." Standard distributed data parallel is the second one, so out of the box you get a bigger batch size, not a bigger model.

Does the second GPU need a full x16 slot?

Not for the default layer split, which only passes a small activation tensor between cards. It matters much more for tensor-parallel serving and for model load times. Check your board rather than your chipset: ASUS's ProArt X870E-Creator WiFi runs two CPU-fed slots at x8/x8, while the ROG Strix X870E-E Gaming WiFi's second x16-length slot is a PCIe 4.0 x4 link from the chipset. At 1.97 GB/s per PCIe 4.0 lane, that is 7.9 GB/s against 31.5 GB/s for a PCIe 5.0 x8 slot.

Can I mix a 3090 and a 3060, or an NVIDIA and an AMD card?

Mixing NVIDIA cards of different sizes works — llama.cpp's --tensor-split takes proportions like 3,1 precisely for this, and the pipeline runs at the pace of the slowest stage. Mixing Radeon cards with different GFX versions is documented too, via per-device HSA_OVERRIDE_GFX_VERSION_0 and _1 variables. Mixing vendors in one pooled model is not a supported path in the mainstream runtimes; treat an NVIDIA card and an AMD card as two separate machines that happen to share a case.

Sources

• llama.cpp server README — --split-mode, --tensor-split, --main-gpu help text
• llama.cpp RPC backend README — distributed inference across machines, and its security warning
• Ollama FAQ and Ollama GPU docs — multi-GPU loading behaviour, CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, HSA_OVERRIDE_GFX_VERSION_n
• vLLM parallelism and scaling — tensor vs pipeline parallelism, NVLink guidance; attention-head constraint from vllm/config/model.py
• ComfyUI comfy/cli_args.py — --cuda-device, --default-device; no model-splitting flag
• ComfyUI-MultiGPU README — "memory management, not parallel processing"
• Hugging Face Accelerate FSDP guide — FSDP sharding strategies mapped to DeepSpeed ZeRO stages
• Unsloth multi-GPU training docs — current support status
• NVIDIA GeForce spec comparison and the 30 Series page — VRAM, memory bandwidth, TDP, minimum system power, NVLink
• ASUS ProArt X870E-Creator WiFi and ROG Strix X870E-E Gaming WiFi — expansion slot lane configurations
• Model file sizes from the published Hugging Face repositories for Llama 3.3 70B, Qwen3 32B, Qwen3-Coder 30B-A3B, Mistral Small 3.2 24B and gpt-oss-120b GGUF builds