eGPU for Local AI: Thunderbolt vs USB4 vs OCuLink

Where do the numbers on this page come from?

Published interconnect specifications, plus arithmetic derived from them with the formula printed next to every result. There are no tokens-per-second measurements here: we do not own an eGPU enclosure, and a borrowed benchmark would only tell you what one laptop did on one runtime version on one afternoon.

The bandwidth maths is the more durable answer. It tells you why the result comes out the way it does, and it stays true when llama.cpp ships a new release.

If you want measured numbers for your own hardware, our benchmark methodology guide shows how to produce them properly — fixed prompt, fixed seed, discard the first run.

Why does an eGPU hurt gaming far more than LLM inference?

Gaming benchmarks fall off a cliff over Thunderbolt because the PCIe bus sits on the render loop's critical path. Every frame pushes draw calls, streams textures, and — critically — has to send the finished frame back across the link to reach the laptop's internal display. That round trip happens sixty or more times per second.

LLM inference has an entirely different traffic profile:

1. Weights load once. A model is copied into VRAM at startup and stays there. After that the link is mostly idle.
2. Per-token traffic is tiny. The host sends a token ID and gets back a probability vector. Measured in kilobytes, not megabytes.
3. The work is on-GPU. Attention and feed-forward matmuls all read from VRAM, which is on the card, on the far side of the link. Making the link slower does not slow them down.
4. There is no frame to send back. Nothing goes to a display at all.

That is the whole argument, and the next three sections put numbers on each part of it.

Thunderbolt 4 vs USB4 vs OCuLink: how much bandwidth do you get?

The headline "40 Gbps" on a Thunderbolt or USB4 port is the total signalling rate shared across display, USB and PCIe traffic. What matters for an eGPU is the PCIe tunnel underneath it. Specifications from the USB Implementers Forum and PCI-SIG:

The PCIe figures come straight from the lane rates: PCIe 3.0 runs 8 GT/s per lane with 128b/130b encoding, giving 0.985 GB/s per lane; PCIe 4.0 doubles that to 1.969 GB/s per lane; PCIe 5.0 doubles it again. Multiply by the lane count. Real transfers land below these ceilings once protocol overhead and the tunnelling layer take their cut — treat every number in the table as an upper bound.

Three practical consequences:

• OCuLink is roughly twice the bandwidth of Thunderbolt 4 and costs less, because an OCuLink dock is essentially a passive PCIe cable adapter with no Thunderbolt controller inside it. Its problem is availability: only certain mini PCs and handhelds expose the port.
• USB4 is not a guarantee. PCIe tunnelling is optional in the USB4 specification. A port can be fully USB4-compliant and still refuse to host an eGPU. Check the host vendor's documentation for the specific machine, not the chipset.
• Plain USB-C carries no PCIe at all. USB-C with DisplayPort alt-mode is a display connector. It cannot host an eGPU under any circumstances.

How long does a model take to load over an eGPU link?

This is the one place the interconnect genuinely bites, and it is straightforward to compute:

Weight footprint at Q4_K_M is about 0.6 GB per billion parameters, so:

Real load times run several times these floors. The point is the ratio: an external link costs you seconds at model-swap time, and the penalty scales with model size, not with how long you then use the model. If you load one model in the morning and chat with it all day, the cost is a rounding error. If you are hot-swapping between four models in a benchmark harness, you will feel it.

Does the link cap your tokens per second?

Work out what actually crosses the bus per generated token.

During generation with the model fully resident on the GPU, the host sends the previous token's ID — a handful of bytes — and receives the output distribution back. Llama 3 has a vocabulary of 128,256 tokens, so a full float32 logits vector is:

That is the whole answer. A link running at roughly one-eighth the bandwidth of an internal PCIe 4.0 x16 slot is being asked to carry about one percent of its capacity. There is no mechanism by which it becomes the bottleneck for generation.

Prompt processing is the case people expect to be different, and it mostly is not — for a fully offloaded model, prefill also runs entirely on the GPU, and the host only sends token IDs. It is a few kilobytes for a long prompt.

Partial CPU offload is the interesting case, and even there the link is rarely the villain. When some layers sit on the CPU, a hidden-state vector crosses the boundary at the split. For an 8B model with a 4,096-dimension hidden state in fp16, that is 8 KB per crossing — at 30 tok/s with two crossings per token, under 0.5 MB/s. What actually makes partial offload slow is that the CPU-resident layers are being computed against system RAM at DDR5 speeds (roughly 80-120 GB/s on a dual-channel desktop) instead of GDDR6X at 1,008 GB/s. That penalty is identical whether the GPU is internal or external.

What actually limits eGPU inference speed?

The card's own VRAM bandwidth, exactly as it would in a desktop slot. Token generation reads the active weights out of VRAM once per token:

An RTX 4090 has 1,008 GB/s of memory bandwidth whether it is bolted into a motherboard or sitting in an enclosure on your desk. On an 8B model at Q4_K_M (4.8 GB) that is a ceiling of roughly 210 tok/s in both cases. The enclosure does not change the number, because the number never depended on the link.

This is why the practical advice inverts the gaming advice: pick the card, not the interconnect. A faster card in an enclosure beats a slower card in a slot for generation work. Our best GPUs for local AI guide covers the card choice, the RTX 5090 vs 4090 comparison works through the same bandwidth arithmetic at the top end, and the used GPU buying guide covers sourcing a 24GB card secondhand.

Where the interconnect does matter for card choice: VRAM capacity is unchanged by an enclosure, so all the usual fit rules apply. Check how much VRAM your models need before you buy either component.

Does an eGPU work on a Mac?

No, on two separate counts, and both are worth understanding because they are not going to change:

1. macOS has had no NVIDIA driver for years. NVIDIA stopped shipping macOS web drivers after macOS 10.13 High Sierra. No CUDA, no NVIDIA GPU acceleration, regardless of how the card is attached.
2. Apple Silicon Macs do not support eGPUs at all. External GPU support was an Intel-Mac feature built on the Intel graphics driver stack. Apple did not carry it forward to M-series machines. An enclosure plugged into an M-series Mac may enumerate as a PCIe device, but no GPU driver will bind to it.

The good news is that Apple Silicon already solves the problem eGPUs exist to solve. Unified memory means a 64GB Mac has 64GB available to the model, which is more than any single consumer NVIDIA card offers. The right move on a Mac is to buy memory, not an enclosure — our Mac local AI setup guide and Apple M4 buying guide work through the tiers.

Which host and enclosure should you buy?

There are only two categories of external GPU hardware, and they differ in what is inside the box:

Sizing the power supply is arithmetic, not opinion. Look up your card's total graphics power on the vendor's spec page — an RTX 4090 is rated at 450W, an RTX 5090 at 575W — then add headroom for transient spikes, which on high-end NVIDIA cards can substantially exceed the rated board power for very short intervals. A 400W enclosure PSU physically cannot deliver 450W to a 4090; that is not a judgement call about margins, it is a number that does not reach. Thunderbolt enclosures publish their PSU rating in the spec sheet, so check it against the card before you buy, not after.

Pairing it with a host. The natural companion to an eGPU is a machine that is otherwise excellent but has no room for a graphics card: a mini PC or a laptop you already own. If you are building around a mini PC, prefer one with a physical OCuLink port over one with only USB4 — see the best mini PCs for Ollama for the current shortlist. If you already own a Thunderbolt 4 laptop, the enclosure route costs you nothing in host hardware, which is usually the point.

If you are building a desktop from scratch, do not buy an eGPU. Put the card in a slot. The enclosure is a solution to "I have a machine with no PCIe slots", and if that is not your problem, you are paying several hundred dollars for a downgrade.

How do you set up an eGPU for Ollama?

The procedure below is the documented path for Linux hosts. Ollama's own GPU documentation is the reference for which cards and driver versions are supported.

Step 1: confirm the host sees the card

# Confirm the GPU enumerated over Thunderbolt / USB4 / OCuLink
lspci -nn | grep -i nvidia
# Should show: VGA compatible controller [0300]: NVIDIA Corporation ...

# Confirm the negotiated PCIe link width and speed
sudo lspci -vvv -s 04:00.0 | grep -i "LnkSta:"
# x4 is expected for TB4 / USB4 / OCuLink.
# Speed 16GT/s = PCIe 4.0 (OCuLink); 8GT/s = PCIe 3.0 (Thunderbolt tunnel).

If the width comes back as x2 rather than x4, you have halved your load bandwidth — usually an older enclosure or a mismarked cable.

Step 2: authorise the Thunderbolt device

Linux will not let a Thunderbolt device function until it is explicitly authorised. This trips up nearly everyone the first time.

boltctl list
boltctl authorize <device-uuid>
boltctl enroll <device-uuid>

OCuLink docks skip this step entirely — the OS sees a normal PCIe device.

Step 3: install the GPU driver

# Ubuntu 24.04
sudo apt install -y nvidia-driver-550 nvidia-cuda-toolkit
sudo reboot

nvidia-smi

Step 4: install Ollama and confirm full offload

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b --verbose "Hello"

The line to look for in the verbose output is the layer offload count — every layer should land on the GPU. If some layers stayed on the CPU, the model did not fit in VRAM, and the speed you measure afterwards is telling you about your system RAM rather than about your eGPU link.

Step 5: check that the link really is idle during generation

sudo nvidia-smi dmon -s u -c 60

The PCIe utilisation column should be low during steady-state generation and rise only while a model is loading. That is the arithmetic from earlier showing up in practice. If it is pinned high during generation, you are almost certainly running with partial CPU offload rather than a fully resident model.

What goes wrong with eGPUs?

A USB4 port that does not tunnel PCIe. The most expensive mistake available, because the port passes every other USB4 test. Verify against the specific machine's documentation before buying an enclosure.

An undersized enclosure PSU. Do the subtraction before you order. A supply rated below the card's total graphics power cannot run it, and the failure mode is a crash under load rather than a refusal to start.

Thunderbolt 3 enclosures negotiating x2. Some pre-2020 enclosures expose only two PCIe lanes. Everything works; your model loads take twice as long. Check the lane count in the product specification.

A cable that is not what it claims. An uncertified or mismarked cable will silently negotiate down. Buy a certified Thunderbolt 4 cable at the length you actually need.

Skipping boltctl on Linux. The device will simply not appear until it is authorised, which reads exactly like a hardware fault.

Sleep and hibernate. Thunderbolt eGPUs generally do not survive suspend cleanly. On a machine you use as an inference server, disable suspend rather than rebuilding the link after every wake.

Hot-unplugging mid-inference. Confirm the GPU is idle before disconnecting. Pulling the cable while the driver holds state can require a reboot to recover.

Buying an NVIDIA card for a Mac. It will not work. See the Mac section above.

Frequently asked questions

How much performance does an eGPU cost you for local AI?

For token generation with the model fully resident in VRAM, the link is carrying roughly one percent of its capacity, so the honest answer is that it is not the limiting factor — the card's own VRAM bandwidth is, and that is unchanged by the enclosure. Where you do pay is model load time, which scales directly with the link's bandwidth: a Thunderbolt 4 tunnel at about 4 GB/s takes roughly twice as long as an OCuLink link at about 7.88 GB/s to move the same weights.

Which interconnect is best for an eGPU AI rig?

OCuLink, if your host has the port. It is a native PCIe 4.0 x4 link with no tunnelling layer, roughly double the bandwidth of Thunderbolt 4 or USB4, and the dock hardware is cheaper because it is essentially a passive adapter. Thunderbolt 4 is the fallback and is far more widely available. Plain USB-C is not an eGPU interface at all.

Is an external RTX 4090 better than an internal RTX 3090 for LLMs?

On the arithmetic, yes for generation: the enclosure does not change the 4090's 1,008 GB/s of VRAM bandwidth, and that is the number that sets the ceiling. The 3090's 936 GB/s is close enough that the gap is smaller than gaming comparisons suggest, so the decision often comes down to price and power draw rather than speed.

Does an eGPU work with a Mac for local AI?

No. macOS has had no NVIDIA driver since macOS 10.13, and Apple Silicon Macs dropped external GPU support entirely — it was an Intel-Mac feature. Buy unified memory instead; a 64GB Mac holds larger models than any single consumer NVIDIA card.

Can I run a 70B model on an eGPU?

Not on one 24GB card. A 70B model at Q4_K_M is around 42GB of weights, so a single 4090 or 3090 needs partial CPU offload, and the offloaded layers run at system-RAM speed regardless of how the GPU is attached. Two 24GB cards or one 48GB card is the clean answer — see our distributed inference guide.

Will plain USB-C work as an eGPU connection?

No. USB-C is a connector shape. Only Thunderbolt 3/4 and USB4 hosts that implement the optional PCIe tunnelling feature can carry PCIe traffic, and OCuLink carries it natively over a different cable entirely.

Does an eGPU need its own power supply?

Yes, always. A GPU draws far more than any host can push down a cable. Thunderbolt enclosures include a PSU; OCuLink docks expect you to connect a standard ATX supply. Check the rating against your card's published total graphics power before you buy.

Why do gaming eGPU benchmarks look so much worse than this?

Because rendering puts the bus on the critical path — textures and draw calls go out every frame, and on a laptop the finished frame has to come back across the link to reach the internal display. Inference does neither. The workloads are not comparable, which is why gaming eGPU reviews are the wrong evidence for this decision.

Conclusion

External GPUs are a genuinely sensible way to add local AI capability to hardware that has no room for a graphics card, and the reason is that LLM inference asks almost nothing of the interconnect after startup. Buy an eGPU to extend a mini PC or a laptop you already own. Do not buy one for a desktop with a free slot, and do not buy one for a Mac.

When you are choosing between an OCuLink dock and a Thunderbolt enclosure, you are choosing between roughly 7.88 GB/s and roughly 4 GB/s of load-time bandwidth — a real difference, but one you pay once per model swap rather than once per token.