CUDA Optimization for Local LLMs: Every Lever, Ranked

Short answer: the highest-impact CUDA optimization is getting the entire model into VRAM — pick a quantization that fits, then set --n-gpu-layers to the full layer count. Everything else is refinement. After that, in order: FlashAttention (the win grows with context length), KV-cache quantization at Q8_0, and FP8/INT4 weights on hardware that supports them. If you serve more than one user, switch to a continuous-batching server; that single change outweighs every kernel flag combined.

This page is the reference for all of it: the exact flags for llama.cpp, Ollama, vLLM and TensorRT-LLM, the arithmetic for sizing a model to a card, and the trade-offs nobody mentions in the README.

On numbers: performance figures here are either attributed to the paper or vendor page that published them, or computed in front of you from published specifications. Where neither is possible, the page tells you how to measure it on your own hardware instead of quoting a figure from somebody else's.

Table of Contents

1. Which CUDA optimizations actually matter?
2. Drivers, CUDA Toolkit, cuDNN, NCCL
3. Which quantization should you use?
4. How do you set --n-gpu-layers correctly?
5. Does FlashAttention help on consumer GPUs?
6. KV-cache quantization & PagedAttention
7. Tensor Cores & mixed precision
8. cuBLAS, cuDNN, and kernel selection
9. CUDA graphs
10. Tensor parallelism, pipeline parallelism, NVLink
11. Speculative decoding & Medusa
12. When do you need continuous batching?
13. Should you power-limit your GPU?
14. MIG, MPS, and multi-tenant isolation
15. Framework-specific tuning
16. Profiling: Nsight and nvidia-smi dmon
17. Common mistakes that silently kill performance
18. Reference configs by GPU
19. FAQ

---

Which CUDA Optimizations Actually Matter?

Before tuning anything, know where the time goes. Single-stream LLM decoding is memory-bandwidth bound: every generated token requires reading the whole active model out of memory. That one fact explains the ranking below.

The arithmetic that drives everything else. Model size at Q4_K_M is roughly 0.6 GB per billion parameters, and the ceiling on decode speed is:

tokens/sec ceiling = memory bandwidth (GB/s) / model size in memory (GB)

An RTX 4090 has 1,008 GB/s of VRAM bandwidth (NVIDIA specification), so a 70B at Q4_K_M (~42 GB) has a ceiling of about 24 tok/s on that card if it fits. Push 18 GB of it into system RAM across PCIe 4.0 x16 — about 31.5 GB/s each way, computed below — and the slow path now sets the pace. That is where the "fit it in VRAM" advice comes from, and it is why it dwarfs every other item on this list. Treat the ceiling as an upper bound: real output lands below it.

The column that is deliberately missing is "typical speedup". Any honest number there depends on your card, model, quant, context length and batch size — so instead of a made-up multiplier, the profiling section shows how to get the real figure for your setup in about two minutes with llama-bench.

If you do only the top three, you have captured most of what is available on a single-user desktop. The rest is fine-tuning.

---

Foundation: Drivers, CUDA Toolkit, cuDNN, NCCL

The fastest kernels in the world cannot save you from a stale driver. Versions matter.

Recommended baseline (May 2026)

# Verify your stack
nvidia-smi --query-gpu=driver_version,name,vbios_version --format=csv
nvcc --version
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.backends.cudnn.version())"

Driver Persistence Mode (Linux)

By default the driver tears down state when no client holds the GPU, so the next process pays for a fresh CUDA context before any work starts. That is a fixed cost per process launch, invisible on a long-running server and very visible on a short CLI invocation. Enable persistence mode for long-running services:

sudo nvidia-smi -pm 1   # Enable persistence (deprecated on newer drivers)
# Modern replacement (driver 470+):
sudo systemctl enable --now nvidia-persistenced

Compute Mode

For dedicated inference servers, set exclusive mode so a single CUDA context owns the GPU and avoids context-switch overhead:

sudo nvidia-smi -c EXCLUSIVE_PROCESS

Set back to default (-c DEFAULT) on workstations where you also game.

---

Which Quantization Should You Use?

Quantization is the single biggest lever — but the right format depends on your GPU generation and framework.

Format compatibility matrix

Practical recommendations

• 8B models on 12-24GB VRAM: FP16 / BF16 is fine; quality is highest, speed is plenty.
• 14-32B models on 24GB VRAM: Q5_K_M (GGUF) or AWQ-INT4. Sweet spot for quality.
• 70B models on 24GB VRAM: Q4_K_M (GGUF) at ~42GB total — partial offload required.
• 70B models on 48GB VRAM (2x 3090, A6000): Q5_K_M or Q4_K_M fully on GPU.
• 70B models on 64-80GB+ (H100, 2x 5090): FP8 or AWQ-INT4 for max speed.

Why BF16 beats FP16 in 2026

BF16 has the same exponent range as FP32 (8 bits) but fewer mantissa bits (7 vs 23). For LLM inference this is almost always a net win: no overflow at long context, minimal quality difference, same throughput as FP16 on Ampere and newer. PyTorch / vLLM / TensorRT-LLM all default to BF16 for new models.

# vLLM example — explicitly request BF16
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", dtype="bfloat16")

FP8 — the 2025-2026 breakthrough

Hopper introduced FP8, Ada brought it to consumer GPUs (RTX 40-series), and Blackwell doubled FP8 throughput again. Two formats:

• E4M3 — 4 exponent bits, 3 mantissa bits — used for weights and activations.
• E5M2 — 5 exponent bits, 2 mantissa bits — used for gradients (training only).

For inference you almost always want E4M3 with per-tensor or per-channel scaling. The mechanical benefit is easy to state: one byte per weight instead of two halves the bytes read per token, which is a direct hit on the bandwidth bound. The quality cost after calibration is small but model-specific — validate it on your own evaluation set before shipping, and read the calibration notes in the TensorRT-LLM and vLLM quantization docs rather than assuming it is free.

# vLLM with FP8 KV cache and FP8 weights (Ada+ required)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --quantization fp8 \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 32768

INT4 via AWQ vs GPTQ vs GGUF

• AWQ (Activation-aware Weight Quantization) — preserves salient weights based on activation magnitude. Best quality at 4-bit. Strongly recommended for vLLM and TensorRT-LLM.
• GPTQ — older but widely available; group-size 128 is standard.
• GGUF Q4_K_M / IQ4_XS — llama.cpp's k-quants and i-quants. IQ-quants use an importance matrix to pack more quality into the same bit budget, at the cost of slower kernels; the llama.cpp repository publishes the perplexity comparisons between the two families.

Sizing a 70B by format — the arithmetic

Weight size is arithmetic, not opinion: multiply the parameter count by the bytes per weight, then add roughly 10% for the parts that stay at higher precision (embeddings, norms, and in GGUF k-quants the more sensitive tensors). For Llama 3.1 70B:

Combine that with the ceiling formula from the ranking section and the practical conclusion falls out without needing anyone's benchmark: on a single 24 GB card a 70B is always a partial offload and therefore always slow, and the fastest 70B configuration you can build at home is two 24 GB cards holding an INT4 or Q4 quant entirely in VRAM.

On quality: the AWQ, GPTQ and llama.cpp k-quant/i-quant projects all publish their own perplexity and benchmark comparisons, and those are the numbers to read — a 4-bit quant's accuracy loss is small but real, and it is format- and model-specific. We do not publish quality scores we have not run.

For multi-GPU setups (2x 4090, 2x 5090), AWQ-INT4 + vLLM is the highest-throughput option for 70B models because it is the combination that both fits and batches.

---

How Do You Set --n-gpu-layers Correctly?

In llama.cpp / Ollama, this single flag controls how many transformer layers run on the GPU. Get it wrong and you lose 5-10x performance.

Layer counts by model

Weight sizes below are computed with the same rule used throughout this page — 0.6 GB per billion parameters at Q4_K_M — so you can check them and extend the table to any model you like. Add roughly 1-2 GB on top for the KV cache and runtime at ordinary context lengths, more as context grows.

Mixtral is the row that catches people out: an MoE model must hold all experts in memory even though only two are active per token, so it sizes like a 47B and decodes like a 13B.

Tuning procedure

# 1. Start with all layers on GPU
./llama-cli -m model.gguf -ngl 999 -c 4096

# 2. If OOM, drop in steps of 4
./llama-cli -m model.gguf -ngl 76 -c 4096   # 70B with 4 layers on CPU
./llama-cli -m model.gguf -ngl 72 -c 4096

# 3. Always keep the output (lm_head) on GPU
./llama-cli -m model.gguf -ngl 72 --override-tensor "output.weight=GPU"

In Ollama

# Modelfile
FROM llama3.1:70b-instruct-q4_K_M
PARAMETER num_gpu 80          # 80 = all 70B layers
PARAMETER num_ctx 8192
PARAMETER num_batch 512

Or at runtime:

OLLAMA_NUM_GPU=80 ollama run llama3.1:70b

Why "all layers" beats "almost all layers"

With even one layer on the CPU, every generated token requires a PCIe round-trip for that layer's weights. The link budget is computable: PCIe 4.0 x16 runs at 16 GT/s per lane with 128b/130b encoding, giving 16 x 16 / 8 x (128/130) = 31.5 GB/s in each direction, versus roughly 1 TB/s inside a modern card's VRAM. That is a ratio of about 32:1, so a handful of offloaded layers can dominate per-token latency even though they are a small fraction of the model. Always size your quantization to fit fully on the GPU if you can.

Multi-GPU tensor split

For multi-GPU setups, control distribution explicitly:

# llama.cpp — proportionally split across two 24GB GPUs
./llama-cli -m model.gguf -ngl 999 --tensor-split 24,24

# Asymmetric: 4090 (24GB) + 3090 (24GB) — keep more on the faster card
./llama-cli -m model.gguf -ngl 999 --tensor-split 28,22

---

Does FlashAttention Help on Consumer GPUs?

Standard attention has O(N²) memory. FlashAttention restructures the computation to O(N) memory by tiling the QKV matmul and keeping a running softmax in SRAM, which also removes most of the HBM round-trips that made the naive version slow.

Versions and hardware support

FA3 adds FP8 support, asynchrony via warp specialization, and better tail handling. The FlashAttention-3 paper reports 1.5-2.0x over FA2 on H100 with FP16, reaching up to 740 TFLOPS/s (about 75% utilization), and close to 1.2 PFLOPS/s with FP8.

Enabling FlashAttention

llama.cpp / Ollama:

# llama.cpp
./llama-cli -m model.gguf -ngl 999 -fa

# Ollama Modelfile
PARAMETER flash_attn true

vLLM:

# Auto-selected; force a specific backend if needed:
VLLM_ATTENTION_BACKEND=FLASH_ATTN vllm serve <model>
# On Hopper/Blackwell, use FlashInfer or FA3:
VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve <model>

TensorRT-LLM: built into the engine; no flag needed.

Why the win grows with context length

The mechanism is in the complexity, and you can reason about it without a benchmark. Standard attention materializes an N x N score matrix, so attention memory grows with the square of sequence length: doubling context quadruples that term. FlashAttention never materializes it — it tiles the computation and keeps the running softmax in SRAM — so the same term grows linearly.

At 2K context, attention is a small slice of the work and the difference is marginal. At 16K it is the dominant memory consumer, and at 32K standard attention frequently will not allocate at all on a consumer card while FlashAttention runs comfortably. That crossover is the practical reason to enable it: not "it is x% faster" but "it is the difference between running and OOM". For RAG and agent workloads, where context routinely passes 8K, treat it as mandatory.

The authors' own measurements are in the FlashAttention-2 paper, which reports roughly 2x over FlashAttention-1 and 50-73% of theoretical peak FLOPs/s on A100. To get the figure for your card and context length, use llama-bench with and without -fa — see profiling.

---

KV-Cache Quantization & PagedAttention

For autoregressive generation, the KV cache is often the dominant memory cost — a Llama 3.1 70B at 32K context with FP16 KV cache eats ~20 GB on its own. Two complementary techniques:

KV-cache quantization

Quantize the K and V tensors in place. Q8_0 is essentially free quality-wise; Q4 is risky.

# llama.cpp — requires FlashAttention
./llama-cli -m model.gguf -ngl 999 -fa \
    --cache-type-k q8_0 --cache-type-v q8_0

# vLLM — FP8 KV cache (Ada+ required for hardware acceleration)
vllm serve <model> --kv-cache-dtype fp8_e4m3

Memory savings: ~50% from FP16 → 8-bit, ~75% from FP16 → 4-bit. On long-context workloads this directly translates to bigger usable contexts or smaller GPU requirements.

PagedAttention (vLLM)

vLLM stores the KV cache in fixed-size blocks (default 16 tokens) instead of one contiguous reservation per sequence. The PagedAttention paper reports that existing systems wasted 60-80% of KV-cache memory to fragmentation and over-reservation, and that vLLM holds waste under 4% — which translates directly into more concurrent requests in the same VRAM. The same paper reports 2-4x higher throughput than FasterTransformer and Orca at matched latency. That memory efficiency, not a faster kernel, is why a paged server beats a single-stream runtime under load.

vllm serve <model> --block-size 16 --gpu-memory-utilization 0.92

Tune --gpu-memory-utilization upward (0.95-0.97) on dedicated inference boxes; leave it at 0.85-0.90 on workstations where you also run other apps.

Prefix caching

Both vLLM and TensorRT-LLM support prefix caching — system prompts and few-shot exemplars are computed once and reused across requests. The saving is proportional to how much of your prompt is a shared prefix: an agent loop that resends a 4,000-token system prompt before 200 tokens of new input skips prefill on 95% of the prompt from the second request onward. Time-to-first-token is where you see it.

vllm serve <model> --enable-prefix-caching

---

Tensor Cores & Mixed Precision

Tensor Cores are specialized matrix-multiply units. Every CUDA generation since Volta (V100) has them, but the supported types changed:

Making sure you actually use Tensor Cores

PyTorch:

import torch
torch.backends.cuda.matmul.allow_tf32 = True       # Ampere+
torch.backends.cudnn.allow_tf32 = True
torch.set_float32_matmul_precision("high")          # alias for TF32 on

For pure inference, dtypes BF16 / FP16 / FP8 / INT8 automatically dispatch to Tensor Cores. FP32 does not.

Mixed precision in custom code

with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    out = model(input_ids)

llama.cpp, Ollama, vLLM, and TensorRT-LLM all already use Tensor Cores correctly when given a compatible dtype.

---

cuBLAS, cuDNN, and Kernel Selection

These are the math libraries underneath every framework. You usually do not touch them directly, but a few flags matter.

cuBLAS LT and heuristics caching

cuBLAS chooses kernels at runtime via heuristics. Stable workloads (same shapes repeated) benefit from caching:

# Enable cuBLAS LT heuristic cache
export CUBLASLT_LOG_LEVEL=0
export CUBLASLT_HEURISTICS_CACHE_PATH=/tmp/cublaslt-cache

cuDNN benchmark mode (PyTorch)

torch.backends.cudnn.benchmark = True   # autotune for fixed-shape workloads

Use only when input shapes are stable (which is true for inference once context length stabilizes). Setting this on dynamically-shaped training can hurt.

llama.cpp build flags

If you build llama.cpp yourself, build with:

cmake -B build \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_F16=ON \
    -DGGML_CUDA_FORCE_MMQ=ON \
    -DGGML_CUDA_FORCE_CUBLAS=OFF \
    -DCMAKE_CUDA_ARCHITECTURES="89;90;120"
cmake --build build -j

CMAKE_CUDA_ARCHITECTURES matters — 89 = Ada (RTX 40), 90 = Hopper, 120 = Blackwell. Building only for your card avoids fat binaries and slightly faster startup. GGML_CUDA_FORCE_MMQ=ON enables custom mat-mul kernels for quantized types that often beat cuBLAS on small batch sizes.

---

CUDA Graphs

A CUDA graph captures an entire sequence of kernel launches and replays them with a single CPU-side operation. For decoding, where the kernel sequence is essentially identical for every token, this removes the per-launch CPU overhead. Whether that is visible depends on the ratio you can work out from a profile: if your kernels are large (big batch, big model) launch overhead disappears into the noise, and if they are small (batch size 1 on a small model) the GPU can end up waiting on the CPU between them. Look for gaps between kernels in an Nsight Systems timeline — that gap is what graphs remove.

Frameworks that use CUDA Graphs

• TensorRT-LLM — yes, automatic
• vLLM — yes, with --enforce-eager false (default off) for decode steps
• llama.cpp — yes since b3000+, automatic when supported
• PyTorch — manual via torch.cuda.graph()

vLLM explicit setting

vllm serve <model> --enforce-eager false   # default is false, but be explicit

Eager mode (--enforce-eager true) disables CUDA graphs — useful for debugging, painful for production. Because the flag is a straight A/B, this is one of the easiest things to quantify for yourself: run your workload both ways and compare.

---

Tensor Parallelism, Pipeline Parallelism, NVLink

For multi-GPU inference, the choice of parallelism strategy is bigger than any kernel-level tuning.

The three strategies

• Tensor Parallelism (TP) — split each matmul across GPUs. Communication: AllReduce per layer. Latency-friendly. Used by vLLM, TensorRT-LLM, DeepSpeed-Inference.
• Pipeline Parallelism (PP) — split the model by layer ranges. Communication: activations between stages. Throughput-friendly for batched workloads, terrible for batch-size-1 latency. Used by llama.cpp, Ollama by default.
• Expert Parallelism (EP) — only for MoE models like Mixtral. Different experts on different GPUs.

When to pick which

vLLM tensor parallel

# 2x RTX 4090
vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
    --tensor-parallel-size 2 \
    --quantization awq \
    --max-model-len 32768

NVLink vs PCIe

PCIe figures below are computed from the standard — lanes x transfer rate x encoding efficiency — so you can check them; NVLink figures are NVIDIA's published totals.

For scale, put those next to on-card VRAM bandwidth: an RTX 4090 moves 1,008 GB/s internally. Even NVLink is an order of magnitude slower than local VRAM, which is why parallelism strategy matters more than interconnect for most home setups.

RTX 4090 and 5090 do not support NVLink. For consumer 70B inference on 2 GPUs, your only options are 2x 3090 with NVLink, or 2x 4090/5090 over PCIe.

NCCL tuning for multi-GPU

export NCCL_P2P_LEVEL=NVL          # require NVLink path if available
export NCCL_DEBUG=WARN
export NCCL_IB_DISABLE=1            # disable InfiniBand on workstations
export NCCL_ASYNC_ERROR_HANDLING=1
export NCCL_NET_GDR_LEVEL=PHB       # GPU Direct RDMA when applicable

For 2 GPUs in one box, defaults are usually fine; the above matters more on 4+ GPU rigs.

---

Speculative Decoding & Medusa

Speculative decoding uses a small "draft" model to guess several tokens, then has the big model verify them in a single forward pass. Net effect: fewer big-model forward passes per generated token.

Methods

Each of those methods publishes its own speedup claims, measured on the authors' hardware with their own draft/target pairing — read them at the source rather than trusting a summary table, because the numbers do not transfer across models. What does transfer is the mechanism: verified speculation cannot change output quality, only latency.

llama.cpp speculative decoding

./llama-cli -m large.gguf \
    --model-draft small.gguf \
    -ngl 999 --draft-max 8 -p "..."

Pair models with the same tokenizer (e.g., Llama 3.1 70B target + Llama 3.2 1B draft).

vLLM speculative decoding

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model meta-llama/Llama-3.2-1B-Instruct \
    --num-speculative-tokens 5

Acceptance rate matters

Speculative decoding only pays off if the draft model usually agrees with the target. Every rejected token is work the big model did and threw away, so a poorly matched draft can leave you slower than no speculation at all. There is no universal cut-off — measure your own acceptance rate with vLLM's --collect-detailed-traces and compare against the same workload with speculation disabled.

---

When Do You Need Continuous Batching?

If you serve more than one user, this single feature is the biggest single throughput win available.

Static batching waits for a batch to fill, then runs all sequences to completion together. Continuous batching swaps in new requests at every decoding step, so the GPU is never idle.

vLLM, TGI (HuggingFace), TensorRT-LLM, and SGLang all implement continuous batching with various names ("iteration-level scheduling," "in-flight batching"). llama.cpp / Ollama do not — they are intended for single-user desktop use.

The reason the gap is so large comes back to the bandwidth bound. Reading the weights is the expensive part of a decode step, and that cost is the same whether you are decoding for one sequence or sixty-four. A single-stream runtime pays it per user; a batching server pays it once and amortizes it across the batch. So aggregate throughput on a single-stream runtime is roughly flat as concurrency rises, while a batching server climbs until it becomes compute- or KV-cache-bound.

The published measurement to anchor on is the vLLM/PagedAttention paper: 2-4x the throughput of FasterTransformer and Orca at the same latency, with the gap widening for longer sequences and larger models. To size it for your own hardware, drive both with the same concurrency using vllm bench serve or a simple async load script and compare tokens/second in aggregate.

Single user, llama.cpp is competitive and far simpler. Multi-user, a batching server is not a tuning choice — it is the architecture. Pick the right tool.

---

Should You Power-Limit Your GPU for Inference?

NVIDIA consumer GPUs ship with aggressive boost behavior. In a long inference run the card heats up, boost clocks drop, and sustained throughput settles below the peak you saw in the first thirty seconds. Capping power trades a little peak for a lower, steadier thermal equilibrium — and because single-stream decode is bandwidth-bound rather than compute-bound, the cost of giving up some core clock is usually smaller than it looks.

How much you give up is card-, cooler- and case-specific, so measure it rather than trusting a number: run llama-bench at stock, apply the cap, run it again, and watch nvidia-smi dmon for the point where tmp stops climbing.

Power limit (Linux)

sudo nvidia-smi -pl 350                # cap RTX 4090 at 350W (stock 450W)
sudo nvidia-smi -pl 280                # cap RTX 3090 at 280W (stock 350W)

Set persistent at boot:

# /etc/systemd/system/nvidia-power-limit.service
[Unit]
Description=NVIDIA GPU power limit
After=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pl 350
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Lock clocks for predictable latency

sudo nvidia-smi -lgc 1500,2520        # lock graphics clock between 1500 and 2520 MHz
sudo nvidia-smi -lmc 10501            # lock memory clock at 10501 MHz (4090 stock)

Locked clocks remove the jitter that boost transitions introduce, which matters when you are measuring latency percentiles for an agent loop rather than average throughput.

Undervolting

On Linux, nvidia-smi constrains voltage indirectly via the -lgc upper bound. On Windows, use the MSI Afterburner or NVIDIA App curve editor: drop the curve, lower the clock ceiling, and validate with a sustained llama-bench run long enough for the card to reach thermal equilibrium — an unstable undervolt fails as a driver reset under load, not in the first minute.

Fan curve

Default fan curves are conservative. For long inference runs, set fans to ramp earlier:

# nvidia-settings (Linux, X required)
nvidia-settings -a "[gpu:0]/GPUFanControlState=1" \
                -a "[fan:0]/GPUTargetFanSpeed=70"

---

MIG, MPS, and Multi-Tenant Isolation

If multiple processes need to share a GPU without one starving the other:

• MPS (Multi-Process Service) — multiplexes CUDA contexts on a single GPU. Works on all NVIDIA GPUs since Volta. Latency-friendly, no isolation.
• MIG (Multi-Instance GPU) — partitions a GPU into hardware-isolated slices. A100, H100, H200 only — not consumer.

Enabling MPS (Linux)

# Per user, before launching CUDA processes
export CUDA_VISIBLE_DEVICES=0
nvidia-cuda-mps-control -d   # start MPS daemon

MPS is useful if you run, e.g., Ollama for chat and an embedding service simultaneously on one GPU — without MPS they serialize CUDA contexts and steal latency from each other.

MIG on H100

sudo nvidia-smi -mig 1
sudo nvidia-smi mig -cgi 19,19,19,19,19,19,19 -C   # seven 1g.10gb instances

For local-LLM hobbyists this is rarely relevant. For shared lab GPUs it is essential.

---

Framework-Specific Tuning

Ollama

# Modelfile
FROM llama3.1:70b-instruct-q4_K_M
PARAMETER num_gpu 80
PARAMETER num_ctx 8192
PARAMETER num_batch 512
PARAMETER num_thread 8
PARAMETER flash_attn true
PARAMETER use_mmap true

Useful environment variables:

export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_KEEP_ALIVE=24h

OLLAMA_KEEP_ALIVE matters — by default Ollama unloads a model after 5 minutes of inactivity, and the next request pays to read the whole thing back off disk. You can estimate that cost: model size divided by your storage's sequential read rate, so a 42 GB 70B on a drive doing 3 GB/s is roughly 14 seconds, and the same model on a SATA SSD at 500 MB/s is well over a minute. Setting 24h is the fix for anything latency-sensitive.

llama.cpp

./llama-server \
    -m model.gguf \
    -ngl 999 \
    -c 8192 \
    -b 2048 -ub 512 \
    -fa \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --no-mmap \
    --threads 8 \
    --tensor-split 24,24

-b 2048 -ub 512 controls logical and physical batch sizes for prompt processing — higher -b is faster on prompt eval, higher -ub uses more VRAM.

vLLM

vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
    --quantization awq \
    --kv-cache-dtype fp8_e4m3 \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-num-batched-tokens 8192

--enable-chunked-prefill interleaves long prompt prefill with decode steps so a 32K-token prompt does not block other requests.

TensorRT-LLM

Build the engine ahead of time, then serve. Engines are GPU-architecture-specific.

# Build (Llama 3.1 70B with AWQ-INT4, TP=2)
trtllm-build \
    --checkpoint_dir ./Llama3.1-70B-awq \
    --output_dir ./engines/llama3.1-70b-awq-tp2 \
    --gemm_plugin auto \
    --gpt_attention_plugin auto \
    --use_paged_context_fmha enable \
    --use_fp8_context_fmha enable \
    --max_input_len 32768 \
    --max_seq_len 33792 \
    --max_batch_size 16 \
    --tp_size 2

# Serve via Triton or trtllm-serve
trtllm-serve ./engines/llama3.1-70b-awq-tp2 --port 8000

ExLlamaV2

Best in class for single-GPU INT4 inference on Ampere/Ada with 24GB-class cards. Use exllamav2_HF loader in text-generation-webui, or the standalone server. EXL2 quantization (variable bit allocation) frequently beats AWQ on quality at the same size.

---

Profiling: Nsight Systems, Nsight Compute, nvidia-smi dmon

You cannot optimize what you do not measure.

Quick health check

nvidia-smi dmon -s pucvmet -d 1

Watch for:

• sm (SM utilization) — low SM utilization with high memory activity is the normal signature of bandwidth-bound decode. Low on both means you are CPU-bound or PCIe-bound, and that is the case worth chasing.
• mem (memory controller activity) — for single-stream decode this is the bottleneck, so it should be the number that is pinned.
• pwr — sitting at your power limit means power is the binding constraint; sitting well under it means something else is.
• tmp — watch whether it is still climbing. Clocks falling while temperature rises is thermal throttling, and it is the reason a benchmark's first thirty seconds flatter the sustained number.

Nsight Systems (timeline)

nsys profile -o llm-trace --stats=true \
    python -c "..."

Open in Nsight Systems UI. Look for gaps between kernels (CPU bottleneck) and unusually long kernels (memory-bound).

Nsight Compute (kernel-level)

ncu --set full -o kernel-report ./llama-cli ...

Heavy hammer; use only when you suspect a specific kernel is slow.

Framework-native profilers

• vLLM: --collect-detailed-traces all writes per-request trace JSON.
• PyTorch: torch.profiler with the tensorboard_trace_handler.
• llama.cpp: llama-bench -m model.gguf -ngl 999 -p 512 -n 128 for repeatable throughput numbers.

---

Common Mistakes That Silently Kill Performance

1. Wrong PCIe slot — second NVMe or chipset PCIe slots often run x4 or x8. Verify with nvidia-smi --query-gpu=pcie.link.width.current,pcie.link.gen.current --format=csv.
2. Resizable BAR off — enable in BIOS. Improves PCIe transfers significantly.
3. Background processes on the GPU — browser hardware acceleration, Discord overlay, OBS. Check with nvidia-smi: every megabyte they hold is a megabyte your model cannot use, and if it pushes you past the point where the model fits, the cost is not a few percent but the whole VRAM-vs-PCIe gap.
4. PCIe ASPM (power saving) — disable in BIOS for inference servers. Link wakeup latency shows up as jitter.
5. malloc / pageable host memory — pageable host memory forces the driver to stage transfers through an internal pinned buffer, so any code path that copies weights or activations via the host should allocate pinned memory instead. Most frameworks do this already; hand-written code often does not.
6. Wrong GGUF quant — IQ-quants are higher quality at the same size but slower; on Ada/Blackwell prefer K-quants for raw speed.
7. OLLAMA_KEEP_ALIVE default of 5 min — model unload + reload kills latency-sensitive workflows. Set to 24h.
8. Persistent mode off — context creation latency at every CUDA process start.
9. Mixing CUDA toolkit versions — match PyTorch's expected CUDA runtime to your driver.
10. Running BF16 model in FP32 — happens silently when frameworks fall back. Always verify with a profile or memory footprint check.

---

Reference Configs by GPU

Model picks below come from the sizing arithmetic — 0.6 GB per billion parameters at Q4_K_M, plus 1-2 GB of headroom — not from a leaderboard. Power-limit values are a starting point for your own A/B, not a measured optimum.

RTX 3090 / 3090 Ti (24 GB, Ampere)

• Fits comfortably: 8B FP16, 14B AWQ, 32B Q5_K_M. A 70B is a partial offload.
• No FP8 hardware, so k-quants and INT4 are your levers.
• llama.cpp with FA2 and Q8 KV cache is the standard 32B configuration.
• Two cards is the only consumer pairing with an NVLink option for tensor parallelism.

RTX 4070 Ti Super / 4080 Super (16 GB, Ada)

• Fits comfortably: 8B BF16, 14B AWQ. A 32B Q4_K_M needs offload or a tighter quant.
• Ada means FP8 is available — enable FP8 KV cache in vLLM for long context.

RTX 4090 (24 GB, Ada)

• Fits comfortably: 8B BF16, 32B AWQ. A 70B is a partial offload.
• 1,008 GB/s of VRAM bandwidth, which is what sets the ceiling on anything that fits.
• vLLM with AWQ weights and FP8 KV cache is the throughput configuration.
• Start power-limit experiments around 350W (stock 450W) and benchmark both ways.

RTX 5090 (32 GB, Blackwell)

• Fits comfortably: 8B and 32B at BF16; a 70B at AWQ-INT4 fits, which 24 GB cards cannot do.
• FA3 and FP8 are both native here, and both target the bandwidth bound directly.
• Lock memory clocks if you see jitter; GDDR7 boost behaviour has been inconsistent on early drivers.

Dual-GPU rigs

• 2x 3090 (48 GB, NVLink available): the cheapest way to hold a 70B Q4 entirely in VRAM, and the only consumer pairing with an NVLink path for tensor parallelism.
• 2x 4090 (48 GB, PCIe only): higher per-card bandwidth and FP8 support, no NVLink, considerably more money.
• 2x 5090 (64 GB, PCIe 5.0): the most headroom, and the only consumer pairing where a 70B fits with real room left for long context.

The choice is mostly a budget question once the model fits, because the moment it fits, per-card VRAM bandwidth is what sets the ceiling.

---

Sources & further reading: llama.cpp discussions | vLLM documentation | TensorRT-LLM repo | FlashAttention-2 paper | FlashAttention-3 paper | PagedAttention / vLLM paper | NVIDIA CUDA Best Practices Guide. Bandwidth and interconnect figures are NVIDIA published specifications or computed from the PCIe standard as shown inline.

---

Frequently Asked Questions

What is the single highest-impact CUDA optimization?

Getting the whole model into VRAM. In llama.cpp and Ollama that means choosing a quantization that fits and setting --n-gpu-layers (or num_gpu) to at least the model's layer count. The reason is the bandwidth gap computed earlier: roughly 1 TB/s inside a modern card versus 31.5 GB/s each way across PCIe 4.0 x16. Every other optimization on this page is a refinement on top of a model that already fits.

Should I use FP16, BF16, FP8, or INT8?

BF16 is the safe default for full precision on Ampere and newer: same exponent range as FP32, so no overflow surprises at long context, and Tensor Core throughput equal to FP16. FP8 (E4M3) is available on Ada, Hopper and Blackwell and halves the bytes read per token, which is exactly the bound that matters — but framework support is uneven (TensorRT-LLM and vLLM yes, llama.cpp no) and it needs calibration. INT8 and INT4 via AWQ or GPTQ are the route when you need the model to be smaller still.

How do I tune --n-gpu-layers correctly?

Start at the model's full layer count (Llama 3 8B has 32, Llama 3 70B has 80, Qwen 2.5 32B has 64), or just use -ngl 999, which llama.cpp clamps to the real count. If it OOMs, drop by 4 at a time until it loads. Keep the output layer and embeddings on the GPU where you can — they are touched on every token. In Ollama the equivalent is PARAMETER num_gpu in the Modelfile.

Does FlashAttention actually help on consumer NVIDIA GPUs?

Yes, and the benefit grows with context length because it turns the O(N²) attention memory term into O(N). At short contexts the difference is marginal; at long contexts it is often the difference between running and failing to allocate. Enable it with -fa in llama.cpp or PARAMETER flash_attn true in Ollama; vLLM and TensorRT-LLM use it by default. FlashAttention-3 adds FP8 and is Hopper/Blackwell only.

Is NVLink worth it for two RTX 3090s?

For tensor-parallel inference it removes a real bottleneck, since NVLink's ~112 GB/s beats PCIe 4.0 x16's 31.5 GB/s per direction for the all-reduce on every layer. For pipeline parallelism — the default in llama.cpp and Ollama — cross-GPU traffic only happens at stage boundaries, so it matters far less. Note that RTX 4090 and 5090 dropped NVLink entirely, so on those cards PCIe is the only option and the question is moot.

Should I power-limit or undervolt for sustained inference?

It is usually worth testing, because sustained throughput is set by the clock the card can hold once it is hot, not by its opening boost. Cap with nvidia-smi -pl <watts> on Linux or the curve editor on Windows, then benchmark long enough for the card to reach equilibrium and compare. How much throughput you lose depends on your specific card and cooling, which is why the right move is an A/B on your own machine rather than copying someone's wattage.

When should I use vLLM or TensorRT-LLM instead of Ollama?

Switch to vLLM the moment you serve more than one user concurrently: PagedAttention plus continuous batching amortize the weight read across the batch, and the vLLM paper reports 2-4x the throughput of the prior generation of servers at matched latency. Choose TensorRT-LLM when single-stream latency is the priority and you can absorb an ahead-of-time engine build per GPU architecture. Stay on Ollama or llama.cpp for single-user desktop work, broad model support and simplicity.

Does KV-cache quantization hurt quality?

Q8_0 is the safe setting and is what most people should use; Q4 KV cache is where users start reporting degradation on long-context retrieval and reasoning, so it is not a good default. Enable it in llama.cpp with --cache-type-k q8_0 --cache-type-v q8_0 (FlashAttention required). Going from 16-bit to 8-bit halves KV memory, which at 32K context and above frees real gigabytes you can spend on a bigger model or a longer window.

Related guides on Local AI Master:

• MLX vs CUDA for Local AI
• Multi-GPU Ollama Setup
• Why Is My Local LLM So Slow?
• Best GPUs for AI 2025
• AWQ vs GPTQ vs GGUF
• Quantization Explained
• VRAM Calculator