Ollama Multi-GPU Setup: Split 70B Models Across 2 GPUs

Short answer: a second GPU gives Ollama more VRAM, not more speed. Ollama runs on llama.cpp, which splits a model by transformer layer across your cards and pushes each token through them in sequence — so two GPUs let you load a 70B model that never fitted before, while single-request tokens per second stays roughly where one card of that class would land. Set OLLAMA_SCHED_SPREAD=1, make both cards visible to CUDA, and load something bigger than 24 GB.

Quick start: with two NVIDIA GPUs already listed by nvidia-smi, set OLLAMA_SCHED_SPREAD=1 and CUDA_VISIBLE_DEVICES=0,1 in the Ollama service environment, restart the service, then run ollama run llama3.3:70b. The model splits across both cards on load.

The official docs cover ollama run. They are much quieter on OLLAMA_SCHED_SPREAD, mixed-VRAM scheduling, and what to do when nvidia-smi shows GPU 1 sitting at zero while GPU 0 saturates. That is what the rest of this page is for.

---

Does a second GPU make Ollama faster?

No — not for a single request. This is the point that causes the most disappointment, so it is worth being precise about the mechanism.

llama.cpp implements pipeline parallelism: layers 0-39 live on GPU 0, layers 40-79 live on GPU 1, and a token walks through GPU 0 then GPU 1. The cards work in sequence, never on the same token at the same time. Contrast that with tensor parallelism (vLLM, TGI, SGLang), where every card holds a slice of every layer and they work on the same token concurrently.

So the per-token time on a two-card pipeline is:

time_per_token = (bytes on GPU 0 / bandwidth of GPU 0)
               + (bytes on GPU 1 / bandwidth of GPU 1)

With two identical cards, that sum collapses to total model bytes / one card's bandwidth — exactly what a single card of the same generation would need if the model fitted on it. Adding a third and fourth card does not change that number either. More cards move the VRAM ceiling; they do not move the bandwidth ceiling.

What multi-GPU is genuinely good for:

1. Models that exceed single-card VRAM. Llama 3.3 70B at Q4_K_M needs roughly 42 GB of weights. Mixtral 8x22B needs about 85 GB. Neither fits on a 24 GB card, and spilling to system RAM costs an order of magnitude in speed because DDR5 bandwidth is a fraction of GDDR6X bandwidth.

2. KV cache headroom for long context. Weights are only part of the bill — see the arithmetic below. Long-context work runs out of VRAM long before the weights do.

3. Concurrent sessions. With OLLAMA_NUM_PARALLEL=2 and two cards you can serve two requests at once, one per card. Aggregate throughput really does scale here even though single-request latency does not. This is the cheapest honest path to a small team server.

For the quantization choices that decide how much VRAM each layer eats, see the AWQ vs GPTQ vs GGUF comparison. To size a budget before buying a second card, use the VRAM requirements for AI models and the per-model Ollama RAM and VRAM table.

---

How much VRAM does a 70B model actually need?

Two numbers, both of which you can check yourself.

Weights. A good rule of thumb for the K-quants is:

VRAM for weights (GB) at Q4_K_M  ~=  0.6 x (parameters in billions)

So 70B lands near 42 GB, 32B near 19 GB, 8B near 4.8 GB. Q5_K_M is roughly 0.75 GB per billion, Q8_0 roughly 1.1 GB per billion, FP16 exactly 2 GB per billion.

KV cache. This is the part people forget, and it is what actually decides your context limit:

bytes per token = 2 (K and V) x kv_heads x head_dim x 2 (FP16) x layers

Llama 3.3 70B uses grouped-query attention with 8 KV heads, head_dim 128, and 80 layers, so:

2 x 8 x 128 x 2 x 80 = 327,680 bytes  ~=  0.33 MB per token

That table explains the single most common multi-GPU complaint: 42 GB of weights "fits" in 48 GB of combined VRAM, then the load fails, because 8K of context plus per-device CUDA workspace pushes the real requirement past what is free. Two 24 GB cards give you 48 GB nominal and meaningfully less usable. Drop context or quantization before blaming the split.

---

Which GPU combinations can run which models?

PCIe lanes. Give each GPU at least PCIe 4.0 x8. Consumer boards with two x16 slots usually drop both to x8 when populated, which is fine — pipeline parallelism moves very little data between cards (the arithmetic is below). An M.2-converted PCIe 3.0 x4 slot is a different story: model load time is bounded by that link, so a 42 GB model takes several minutes to place.

Power. Size the PSU from NVIDIA's published total graphics power, not from guesswork: the RTX 3090 is rated at 350 W and the RTX 4090 at 450 W. Two 3090s is 700 W of board power, two 4090s is 900 W, four 3090s is 1,400 W — before CPU, drives and transient headroom. Specifications are on the NVIDIA GeForce specification comparison.

For full build planning, walk through the budget local AI machine guide.

---

How do I set up Ollama for multiple GPUs?

Step 1: confirm the driver sees every card

# Every GPU should appear with a non-zero memory total
nvidia-smi

# Driver 550.54.14 or newer is the safe floor for mixing Ada and Ampere

If only one GPU appears, the second card is not seated, not powered, or in a disabled BIOS slot. Ollama cannot work around hardware the driver cannot enumerate.

Step 2: install or update Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

Multi-GPU scheduling improved substantially in the 0.1.40 line. Older builds could discover a second GPU and then never assign layers to it on asymmetric-VRAM systems.

Step 3: configure the systemd service

sudo systemctl edit ollama.service

[Service]
Environment="CUDA_VISIBLE_DEVICES=0,1"
Environment="OLLAMA_SCHED_SPREAD=1"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo journalctl -u ollama -f

The journal names every GPU it offloads to when a model loads. If it names only one, jump to the troubleshooting section.

Step 4: pull something big enough to split

# Anything under ~30B fits a single 24GB card and will not split
ollama pull llama3.3:70b
ollama pull mixtral:8x22b

For choosing what deserves the VRAM, the best Ollama models shortlist is the next stop.

---

How do I control which layers go on which GPU?

Three controls cover almost every real rig.

CUDA_VISIBLE_DEVICES

Limits which GPUs Ollama can see; the order is the order of preference.

export CUDA_VISIBLE_DEVICES=0,1     # both cards, GPU 0 first
export CUDA_VISIBLE_DEVICES=1       # pin a model to the second card only
export CUDA_VISIBLE_DEVICES=0,1,3   # four cards, skip the one driving the display

OLLAMA_SCHED_SPREAD

The default packs GPU 0 before touching GPU 1. Setting this to 1 spreads layers across all visible devices instead.

export OLLAMA_SCHED_SPREAD=1   # even spread — symmetric rigs
unset OLLAMA_SCHED_SPREAD      # greedy fill — asymmetric rigs

num_gpu per model

Inside a Modelfile you can pin how many layers go to the primary GPU. This is the escape hatch when automatic placement gets it wrong.

FROM llama3.3:70b
PARAMETER num_gpu 50
PARAMETER num_ctx 8192
PARAMETER num_batch 512

ollama create llama3-multi -f Modelfile
ollama run llama3-multi

Llama 3.3 70B has 80 transformer layers, so num_gpu 50 places 50 on GPU 0 and the remaining 30 on GPU 1 — roughly the ratio you want for a 24 GB + 16 GB pair.

Confirming the split

nvidia-smi dmon -s u -c 30
watch -n 0.5 nvidia-smi

A split model shows both GPUs cycling in lockstep as the token walks the pipeline. If GPU 1 holds zero allocated memory while GPU 0 saturates, no split happened.

llama.cpp documents the underlying split modes in the tensor-split discussion thread, and the environment variables Ollama exposes are listed in the official Ollama FAQ.

---

How fast can a multi-GPU rig actually go?

Token generation is memory-bandwidth bound: to emit one token the GPU must read every weight once. That gives a hard arithmetic ceiling you can compute for any rig, with no benchmark required:

throughput ceiling (tok/s) = memory bandwidth (GB/s) / model size (GB)

This is an upper bound, not a prediction. Attention overhead, sampling, prefill, kernel launches and pipeline hand-offs all cost time, so real output lands well below the ceiling. What the bound is good for is ranking rigs and sanity-checking any number you read elsewhere — a claim above the ceiling is wrong, full stop.

Applied to Llama 3.3 70B Q4_K_M (~42 GB of weights), using each card's published memory bandwidth:

Bandwidth figures are the manufacturer specifications catalogued in the TechPowerUp GPU database.

Read the 4x RTX 3090 row against the 2x RTX 3090 row: four cards have exactly the same single-request ceiling as two. The extra 48 GB buys context and larger models, and nothing else, because the pipeline still reads 42 GB of weights through one card's bus at a time. That single row is the whole argument of this page.

The mixed 4090 + 3090 row is computed as 21/1008 + 21/936 seconds per token, splitting weights evenly; the asymmetric split section below explains how to shift that in the 4090's favour.

If single-request speed is what you need, pipeline parallelism is the wrong tool — tensor-parallel servers work every card on the same token, so their ceiling scales with combined bandwidth rather than one card's. vLLM, TGI and SGLang all do this. The cost is a substantially heavier deployment; a common compromise is to run vLLM behind LiteLLM so your applications keep one OpenAI-compatible endpoint.

---

Does NVLink make Ollama faster?

Barely, and the arithmetic shows why.

Under pipeline parallelism, the only thing crossing the bus per token is the hidden-state vector at the layer boundary. Llama 3.3 70B has hidden_size 8192, so in FP16:

8192 values x 2 bytes = 16,384 bytes  ~=  16 KB per hand-off
at 30 tok/s  =  ~0.5 MB/s of inter-GPU traffic

Now compare the links:

Inter-GPU traffic during generation is about five orders of magnitude below what PCIe 4.0 x16 already provides. Upgrading a link that is 63,000x oversupplied to one that is 225,000x oversupplied cannot move generation speed, because the link was never the constraint.

Where NVLink does earn its price:

• Model load and weight placement, which is bulk transfer (a one-time cost per load, not per token)
• Tensor-parallel frameworks such as vLLM and TGI, where activations really do move in bulk every layer
• Prefill of very long prompts, where the activation tensors are batch-sized rather than one token wide

Where it does not:

• Token-by-token Ollama generation
• Short prompts with long generations
• Mixed CPU/GPU offload, where the PCIe host link dominates anyway

Put the NVLink money toward a third card or more system RAM.

---

Can I mix an RTX 4090 and an RTX 3090?

Yes, and it is a common pairing: a new 4090 alongside the 3090 from the previous build.

The 4090 is less far ahead than the spec sheet suggests

Marketing compares FP16 and FP8 tensor throughput, where the 4090 leads by a wide margin. Token generation does not care: it is bounded by memory bandwidth, and there the published figures are 1,008 GB/s for the 4090 against 936 GB/s for the 3090 — about 8 percent apart. Prefill, which is compute bound, favours the 4090 much more; generation barely notices. Weight the layers by bandwidth, not by TFLOPS, and do not expect the pair to behave like a 4090 and a bystander.

Weight the split toward the faster card

cat > Modelfile.l3-asym <<'EOF'
FROM llama3.3:70b
PARAMETER num_gpu 44
PARAMETER num_ctx 8192
EOF

ollama create llama3-asym -f Modelfile.l3-asym

Both cards hold 24 GB, so the split is constrained by VRAM long before it is constrained by bandwidth; 44 of 80 layers on the 4090 is about as far as you can push it once the KV cache is allocated. The gain over an even split is bounded by that 8 percent bandwidth gap — worth taking, not worth agonising over.

Genuinely asymmetric VRAM (24 GB + 16 GB)

Leave OLLAMA_SCHED_SPREAD unset so the greedy default fills the larger card first. If GPU 1 hits OOM during load, dial num_gpu down two to four layers at a time until it is stable.

Mixing generations

Ada (4090, 4080) with Ampere (3090, 3080) works. Hopper with Ada works. Mixing Turing (RTX 2080) with Ampere or newer is where it gets unpredictable — the older compute capability can force fallback kernels for quantized matmuls. Keep every card at compute capability 8.0 or above.

---

Why is my second GPU sitting at 0% usage?

The model fits on one card

Ollama will not split a model that fits in a single device. Check the size with ollama show llama3.3:70b; if it is under your first card's free VRAM, this is correct behaviour, not a bug.

The environment disagrees with itself

# See what the service actually received, not what your shell exports
sudo systemctl show ollama | grep -E "Environment|CUDA"

sudo systemctl edit ollama
# add: Environment="OLLAMA_SCHED_SPREAD=1"
sudo systemctl restart ollama

A CUDA_VISIBLE_DEVICES set in your interactive shell does nothing for a systemd-managed daemon, which is the single most common cause of a cold second card.

OOM despite "enough" combined VRAM

A 70B model with 42 GB of weights failing on two 24 GB cards is the KV-cache arithmetic from earlier catching up with you: each device also carries context buffers and CUDA workspace, so combined usable VRAM is meaningfully below the nameplate 48 GB.

ollama run llama3.3:70b --ctx-size 8192
ollama pull llama3.3:70b-instruct-q3_K_M

Split is slower than a single card

If the model fits on one card, run it on one card. Forcing a split adds a pipeline hand-off per token for no benefit. Remove OLLAMA_SCHED_SPREAD=1, or pin with PARAMETER num_gpu 999 to keep everything on GPU 0.

Crashes part-way through a generation

Cap board power to reduce transient spikes, which is the cheapest thing to rule out before you start suspecting the driver:

sudo nvidia-smi -i 0 -pl 320
sudo nvidia-smi -i 1 -pl 320

nvidia-smi dmon -s pucvmt -c 60 > thermals.log

Read thermals.log for sustained clock drops and power-limit flags — those tell you whether the card throttled or the supply sagged.

Different output between one-GPU and two-GPU runs

Floating-point reduction order changes with layer placement, so identical prompts and seeds can diverge. This is expected and harmless in normal use, but it breaks deterministic tests. Pin regression tests to a single GPU.

The Ollama troubleshooting guide covers the single-GPU failure modes exhaustively.

---

Frequently asked questions

Does Ollama support tensor parallelism across multiple GPUs?

Not in the vLLM sense. Ollama uses llama.cpp, which performs layer-level splitting (pipeline parallelism) rather than row/column tensor parallelism. The model is sharded by transformer layer and tokens flow through the cards in sequence, so two GPUs double available VRAM, not throughput. For true tensor parallelism, run vLLM or TGI.

How does Ollama decide which layers go on which GPU?

With num_gpu unset, llama.cpp counts free VRAM on each visible CUDA device and packs layers in order until that device is full. Given a 24 GB and a 16 GB card, the larger one receives more layers. Override with OLLAMA_SCHED_SPREAD=1 to spread evenly, or pin the count per model in a Modelfile.

Will NVLink make Ollama faster on dual 3090s?

Effectively no. Pipeline parallelism moves roughly 16 KB of hidden state per token between cards — about 0.5 MB/s at 30 tok/s, some five orders of magnitude below what PCIe 4.0 x16 already delivers. NVLink helps model load time and tensor-parallel frameworks, not token-by-token Ollama generation.

Can I mix an RTX 4090 and an RTX 3090 for one model?

Yes, and it works well. Ollama treats them as separate CUDA devices and assigns layers by free VRAM. Q4_K_M and Q5_K_M behave identically on both because they are dequantized inside the kernel. Generation speed sits between the two cards and closer than you would expect: their published memory bandwidths are only about 8 percent apart.

How much VRAM do I need for Llama 3.3 70B across two GPUs?

About 42 GB for Q4_K_M weights (0.6 GB per billion parameters), plus 0.33 MB per token of KV cache — roughly 2.7 GB at 8K context and 10.7 GB at 32K. Two 24 GB cards handle 8K comfortably; 32K wants 96 GB of combined VRAM or a tighter quantization.

Is multi-GPU worth it over a single A6000 or RTX 6000 Ada?

It depends on what you want from the VRAM. One large card avoids the pipeline hand-off entirely and holds more KV cache; two used 3090s cost less per gigabyte and let you run two different models simultaneously, one per card, which is useful for an embedding plus chat workload. Capacity for one big model, or flexibility for several — pick the constraint you actually have.

Does Ollama work with AMD multi-GPU?

Yes, via the ROCm build. Set HSA_VISIBLE_DEVICES instead of CUDA_VISIBLE_DEVICES and confirm rocm-smi reports every card. Layer splitting behaves the same way as on NVIDIA, though the AMD path has more rough edges and often wants a build from source for recent ROCm fixes.

---

The one-line version

Multi-GPU Ollama is VRAM expansion, not compute expansion. Two 4090s give you 48 GB of fast memory and access to a class of model that fits nowhere else on consumer hardware. They do not give you twice the tokens per second, and the bandwidth arithmetic above says they never will.

Build the rig. Set OLLAMA_SCHED_SPREAD=1. Weight the layers if your cards differ. Watch nvidia-smi dmon until both GPUs cycle together. Then stop thinking about hardware and go build the thing that needed a 70B model in the first place.