Benchmark Your Local AI Setup: tok/s, TTFT and Memory

Published February 12, 2026 • Updated August 2026 • 18 min read

The fastest honest benchmark of a local LLM is one command: ollama run llama3.1:8b --verbose "Write 200 words on caching." The eval rate line it prints is your generation speed in tokens per second, and prompt eval rate is your prompt-processing speed. That single observation is a floor, not a benchmark — this guide is how to turn it into a number that survives someone else trying to reproduce it.

---

What you will measure:

• Generation throughput (tokens/sec) under steady-state load
• Time-to-first-token (TTFT) — the latency users actually feel
• Prompt processing speed (prefill tokens/sec) for long-context workloads
• VRAM and unified-memory headroom under real prompts
• Concurrency curves — at what request rate does throughput collapse

Most "Ollama is slow" or "my GPU is faster than yours" arguments online are unfalsifiable because nobody publishes the prompt, the seed, the context length, the quantization, or the warmup state. This guide closes each of those holes. If you also want to know which models are even worth benchmarking on your hardware, start with the best Ollama models guide and our hardware requirements overview before running anything heavy.

Table of Contents

1. Why Most Local AI Benchmarks Are Wrong
2. The Five Numbers That Actually Matter
3. Setting Up a Clean Test Environment
4. Benchmarking Ollama
5. Benchmarking llama.cpp Directly
6. Benchmarking vLLM for Concurrency
7. How do I know my number is plausible?
8. Common Pitfalls That Tank Numbers
9. Frequently Asked Questions

---

Why Most Local AI Benchmarks Are Wrong

Open any benchmark thread and you will find the same four mistakes:

1. Cold-start measurements. First run loads weights from disk and recompiles kernels. That number is meaningless.
2. Mixed quantizations. Comparing Q4_K_M against Q8_0 is comparing different models, not different machines.
3. Different prompts. A 4-token prompt and a 4,000-token prompt produce wildly different prefill rates.
4. No concurrency control. Single-stream tokens/sec is not the same as multi-user throughput, and most home rigs are tested at concurrency 1.

A defensible benchmark documents the model, the quantization, the context window, the prompt, the temperature, the seed (where supported), the runner version, the hardware, and whether the run was warmed up. We will hit every one of those.

For a deeper philosophical dig into evaluation methodology, llama.cpp's llama-bench tool is the reference implementation most other benchmarks copy from.

---

The Five Numbers That Actually Matter

Forget MMLU. We are measuring the rig, not the model. The five numbers that matter for a local deployment:

1. Generation tokens/sec (eval rate)

Steady-state output speed once the model is generating. This is the number you put on a slide.

2. Time-to-first-token (TTFT)

Wall clock from request to first emitted token. Dominates perceived latency for short prompts and chat UX.

3. Prompt processing tokens/sec (prefill rate)

How fast the model can ingest the prompt. Critical for RAG, long-context coding agents, and document summarization.

4. Effective concurrent throughput

Tokens/sec across N concurrent streams. Single-user vs ten-user numbers can differ 4x in either direction.

5. Peak resident memory

VRAM (GPU) or RSS (CPU/Apple Silicon) at full context. Drives model selection and headroom planning.

---

Setting Up a Clean Test Environment

Before any number is trustworthy:

# 1. Pin the model and quantization explicitly
ollama pull llama3.1:8b-instruct-q4_K_M

# 2. Record the runner version
ollama --version
# llama.cpp:
./llama-cli --version
# vLLM:
python -c "import vllm; print(vllm.__version__)"

# 3. Lock GPU clocks (NVIDIA) so thermals do not skew results
sudo nvidia-smi -pm 1
sudo nvidia-smi --lock-gpu-clocks=1410,1410   # adjust per card
sudo nvidia-smi --lock-memory-clocks=10501,10501

# 4. Drop filesystem cache between runs (Linux)
sync && sudo sysctl -w vm.drop_caches=3

# 5. Disable background indexers (macOS example)
sudo mdutil -a -i off

Standard benchmark prompts

Fix three prompts and reuse them across every run and every machine, so results stay comparable:

SHORT (about 32 tokens):
"Explain in two sentences why local LLMs reduce egress costs versus hosted APIs."

MEDIUM (about 512 tokens):
[paste a Wikipedia paragraph, then ask] "Summarize the above in five bullets."

LONG (about 4,096 tokens):
[paste a long technical doc] "Extract every numeric claim with its source sentence."

Lock temperature, top-p, seed:

export OLLAMA_KEEP_ALIVE=30m   # keep model resident between runs
ollama run llama3.1:8b-instruct-q4_K_M \
  --verbose \
  --temperature 0 \
  --top-p 1 \
  --seed 42

---

Benchmarking Ollama

Step 1: Warm up

# Burn one run to warm caches and JIT
ollama run llama3.1:8b-instruct-q4_K_M --verbose "warmup" > /dev/null

Step 2: Capture verbose stats

ollama run llama3.1:8b-instruct-q4_K_M --verbose \
  "Explain in two sentences why local LLMs reduce egress costs versus hosted APIs."

At the bottom you get a stats block in this shape. The values below are placeholders showing the format — yours will differ, and nothing here is a measurement of any particular machine:

total duration:       <seconds>
load duration:        <ms>
prompt eval count:    <n> token(s)
prompt eval duration: <ms>
prompt eval rate:     <n> tokens/s      <-- prefill
eval count:           <n> token(s)
eval duration:        <seconds>
eval rate:            <n> tokens/s      <-- generation

The two numbers to record are prompt eval rate (prefill) and eval rate (generation). Record load duration too: if it is not near zero, the model was not resident and the run is a cold start you should discard.

Step 3: Automate with the API and measure TTFT

Ollama emits SSE chunks. The first non-empty chunk is your TTFT.

# bench-ollama.sh
#!/usr/bin/env bash
set -euo pipefail

MODEL="${1:-llama3.1:8b-instruct-q4_K_M}"
PROMPT="${2:-Write a 200 word essay on caching strategies.}"
URL="http://127.0.0.1:11434/api/generate"

start=$(date +%s%N)
first_token_ns=""
total_tokens=0

curl -sN -X POST "$URL" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"prompt\":\"$PROMPT\",\"stream\":true,\"options\":{\"temperature\":0,\"seed\":42}}" \
| while IFS= read -r line; do
    if [[ -z "$first_token_ns" ]] && echo "$line" | grep -q '"response":"[^"]'; then
      first_token_ns=$(date +%s%N)
      ttft_ms=$(( (first_token_ns - start) / 1000000 ))
      echo "TTFT: $ttft_ms ms"
    fi
    total_tokens=$((total_tokens + 1))
done

end=$(date +%s%N)
elapsed_s=$(echo "scale=3; ($end - $start) / 1000000000" | bc)
echo "Wall: $elapsed_s s, chunks: $total_tokens"

Run five iterations, drop the first, average the rest.

Step 4: Concurrency

# Hit Ollama with N parallel streams
seq 1 8 | xargs -P 8 -I {} ./bench-ollama.sh llama3.1:8b-instruct-q4_K_M

Compare aggregate tokens/sec against single-stream. Sweep the concurrency level upward until aggregate throughput stops rising — that inflection point is the only capacity number that means anything for your hardware, and it differs enough between cards, runners and quantizations that a figure quoted from someone else's rig is not transferable.

---

Benchmarking llama.cpp Directly

llama.cpp ships a purpose-built benchmark that is more rigorous than wrapping the CLI:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

# Standard sweep: prompt processing 512 / generation 128
./build/bin/llama-bench \
  -m /models/Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf \
  -p 512 -n 128 -t 8 -ngl 99 -r 5

Output has this shape (column values are placeholders, not results from any machine):

| model             |     size | params | backend | ngl |   test |          t/s |
| Meta-Llama-3.1-8B | 4.58 GiB | 8.03B  | CUDA    |  99 | pp 512 | <n> ± <sd>   |
| Meta-Llama-3.1-8B | 4.58 GiB | 8.03B  | CUDA    |  99 | tg 128 | <n> ± <sd>   |

pp is prefill. tg is generation. The ± is one standard deviation across -r 5 runs — publish that, not a single number. This is the format most public comparisons copy, so reporting in it makes your results directly comparable to other people's.

Useful sweeps

# Sweep context lengths to find where prefill collapses
./build/bin/llama-bench -m model.gguf -p 128,512,2048,8192 -n 64

# Compare quantizations on the same hardware
./build/bin/llama-bench \
  -m llama-3.1-8b.Q4_K_M.gguf \
  -m llama-3.1-8b.Q5_K_M.gguf \
  -m llama-3.1-8b.Q8_0.gguf \
  -p 512 -n 128

If you are choosing between quants, our GGUF, AWQ and GPTQ comparison walks through the quality vs throughput tradeoff in detail.

---

Benchmarking vLLM for Concurrency

Ollama and llama.cpp are excellent single-tenant runners. If you are serving multiple users, vLLM's continuous batching changes the shape of the curve completely.

pip install "vllm>=0.6.4"

# Serve
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

vLLM ships its own benchmark harness:

git clone https://github.com/vllm-project/vllm
cd vllm/benchmarks

# ShareGPT-style realistic load
python benchmark_serving.py \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 500 \
  --request-rate 8

The report it prints is what you actually want for capacity planning (fields shown, values elided — fill these in from your own run):

Successful requests:                     <n>
Benchmark duration (s):                  <n>
Total input tokens:                      <n>
Total generated tokens:                  <n>
Request throughput (req/s):              <n>
Output token throughput (tok/s):         <n>
Mean TTFT (ms):                          <n>
P99 TTFT (ms):                           <n>
Mean TPOT (ms):                          <n>

TPOT (time per output token) is vLLM's preferred per-token latency metric — useful when comparing against batched APIs. The reason vLLM's aggregate throughput pulls away from single-tenant runners under load is continuous batching plus PagedAttention, described in the PagedAttention paper (Kwon et al., SOSP 2023). Read that before assuming a concurrency result from one runner transfers to another.

---

How do I know my number is plausible?

You now have a measurement. Before you publish it, check it against physics — this is the step almost nobody does, and it catches both misconfigured rigs and other people's inflated claims.

The arithmetic ceiling

Single-stream generation is memory-bandwidth bound. Producing one token requires streaming every weight through the compute units once, so:

tokens/sec ceiling = memory bandwidth (GB/s) ÷ model size (GB)

Model size at Q4_K_M is roughly 0.6 GB per billion parameters, so an 8B lands near 4.8 GB. Divide your platform's published memory bandwidth by that and you get a hard upper bound:

Read these as ceilings, never as predictions. Attention over the KV cache, sampling, kernel launch overhead and imperfect memory access all cost time the formula ignores, so a real run always lands below its row. Two failure modes this catches:

• Your number is far below the ceiling. Something is throttling, offloading to CPU, or reloading the model each iteration. Go to the pitfalls section below.
• Someone's published number is above the ceiling. Either they batched (which the formula does not cover), used a smaller quant than they claimed, or made it up.

The reporting template

Fill this in and your result is reproducible. Anything missing is a hole someone will poke:

If you are weighing Apple Silicon against discrete GPUs before buying, run this template on both and compare like for like — our Mac Studio vs PC build comparison covers which tradeoffs matter beyond throughput, and AI models for 16GB RAM works the same bandwidth arithmetic for memory-constrained machines.

---

Common Pitfalls That Tank Numbers

1. Forgetting OLLAMA_KEEP_ALIVE

Default is 5 minutes. If your benchmark loop sleeps longer than that, the model unloads and the next "run" pays the load tax. Set OLLAMA_KEEP_ALIVE=30m or longer.

2. Background processes on the GPU

Anything else using the card — a browser decoding video, a display compositor, another model still resident — competes for the same bandwidth your benchmark is measuring. Run nvidia-smi and confirm the only process on the card is your runner.

3. Power and thermal throttling

Laptops and small-form-factor builds hit thermal walls fast, and a throttled run silently reports a lower number with no error. Do not guess at a temperature threshold — ask the driver directly and discard any run where it reports a throttle:

nvidia-smi --query-gpu=clocks_throttle_reasons.active,temperature.gpu,power.draw \
  --format=csv -l 1

On Apple Silicon, sudo powermetrics --samplers thermal -i1000 reports the thermal pressure level; reject runs that leave the nominal state.

4. Mismatched context windows

If you build the GGUF with -c 2048 and llama.cpp with -c 8192, you are benchmarking different memory profiles. Pin --ctx-size explicitly.

5. Ignoring P99 latency

Mean TTFT looks great until one user gets a 3-second wait. Always publish P50 and P99.

6. Single-run reporting

Run at least 5 iterations, drop the first (warmup), report mean and standard deviation. A single number is not a benchmark; it is an anecdote.

7. Not pinning the runner version

llama.cpp ships breaking performance changes monthly. Pin the commit:

git -C llama.cpp rev-parse --short HEAD

Include that hash in your results table.

---

Putting It All Together: The Header Block

Every benchmark you publish should open with a header block like this — a blank form to fill from your own run, not results:

Hardware:        <GPU> / <CPU> / <RAM type and speed>
OS:              <distro + kernel>, <GPU driver version>
Runner:          <ollama|llama.cpp|vllm> <version or commit hash>
Model:           <exact model name>
Quantization:    <quant type> (<file size>)
Context:         --ctx-size <n>
Prompts:         <n> fixed prompts, token counts listed
Sampling:        temperature <n>, top-p <n>, seed <n>
Iterations:      <n> per condition, first dropped as warmup
Concurrency:     <levels tested>
Power state:     <clocks locked?> <throttle observed?>

That is the format auditors, hiring managers, and procurement teams take seriously, because every line closes off an objection. A result without this header is an anecdote with a decimal point.

---

Frequently Asked Questions

Q: Which single number should I report if I only have time for one?

A: Generation tokens/sec at concurrency 1 with a fixed 512-token prompt and 128-token output, averaged over 5 runs after a warmup. It is not the whole story, but it is the least lying number you can report in one figure.

Q: How many runs is enough?

A: Five iterations, dropping the first, is the floor for stable means. For latency P99, you need at least 100 requests because P99 by definition needs ~100 samples to even exist.

Q: Why does my Ollama benchmark show different numbers than llama-bench on the same model?

A: Ollama applies a default system prompt, default sampler settings, and may chunk differently. Match settings explicitly: same context size, same temperature, same prompt, same quantization file.

Q: Should I benchmark with batch size > 1?

A: Only if your real workload uses batched requests. For chat UIs, single stream and concurrency curves are what matter. For offline pipelines, increase batch until VRAM saturates.

Q: How do I measure VRAM properly?

A: nvidia-smi --query-gpu=memory.used --format=csv -l 1 during the run, take the peak. On Apple Silicon, sudo powermetrics --samplers gpu_power plus memory_pressure gives you the equivalent.

Q: My CPU-only run is faster than my GPU run for tiny models. Why?

A: Sub-3B models often fit in CPU cache, and PCIe transfer overhead can dominate GPU runtime. This is real. For models under ~3B parameters, benchmark both backends and pick the winner.

Q: Can I trust hosted leaderboards for my hardware decision?

A: Use them as a sanity check, never as the deciding number. Your prompt distribution, context length, and concurrency profile dictate which runner wins on your machine.

Q: My tokens/sec is far below what the hardware should manage. What now?

A: Divide your platform's published memory bandwidth by the model file's size in GB — that is the ceiling. If you are landing at a small fraction of it, the usual causes in order of likelihood are a partial CPU offload, the model reloading between iterations, and thermal throttling. The sanity-check section has the table and the diagnosis order.

Q: How many concurrent streams should I test?

A: At minimum 1, 4, 8, and 16, then keep doubling until aggregate throughput stops improving. Continuous-batching servers scale further than single-tenant runners, but where your own curve turns over depends on the card, the quantization and the context length. Measure it rather than inheriting someone else's figure.

Q: How often should I re-benchmark?

A: Every minor llama.cpp / vLLM / Ollama bump, every driver update, and after every model swap. Pin the version in your report.

---

Conclusion

A benchmark is only useful if someone else can reproduce it. Pin the model, pin the quantization, pin the runner version, warm up, run five times, publish mean and standard deviation, include P50 and P99 latency, disclose your hardware, and check the result against the bandwidth ceiling before you publish it. Do that once and you will never have to argue about local AI performance with a stranger again — you will just send them your table.

If you are about to make a hardware purchase based on these numbers, pair this guide with our hardware requirements walkthrough and the budget local AI machine build before you click order.

---

New runner releases change these numbers more often than new hardware does. Subscribe to the LocalAimaster newsletter to get told when a version bump is worth re-benchmarking for.