Speculative Decoding Complete Guide (2026): Draft Models, EAGLE, Medusa, n-grams

Speculative decoding is the highest-ROI inference optimization for single-stream LLM serving in 2026 — 2-4x speedup with mathematically identical output to greedy or sampled decoding. The trick: a small fast model proposes K tokens, the large target model verifies all K in a single forward pass, and accepted tokens are free. LLM inference is memory-bandwidth bound, so verifying K tokens costs roughly the same wall-time as generating 1 — when the draft is right, you skip K-1 forward passes for free.

This guide covers every modern speculation technique — separate draft models, EAGLE-2 / EAGLE-3, Medusa, n-gram / prompt-lookup decoding, and Multi-Token Prediction (MTP) — with real configuration for vLLM, SGLang, TensorRT-LLM, and llama.cpp. Includes acceptance-rate tuning, when speculation hurts, batched-serving caveats, and decision trees for picking the right technique per workload.

Table of Contents

1. Why Speculative Decoding Works (Bandwidth Bottleneck)
2. The Core Algorithm
3. Draft Model Speculation
4. EAGLE-2 and EAGLE-3
5. Medusa Heads
6. n-gram / Prompt-Lookup Decoding
7. Multi-Token Prediction (MTP)
8. vLLM Configuration
9. SGLang Configuration
10. TensorRT-LLM Configuration
11. llama.cpp Speculative Mode
12. Tuning Acceptance Rate
13. Sampling vs Greedy: Speedup Trade-off
14. Batched Serving Caveats
15. When Speculation Hurts
16. Decision Tree by Workload
17. Published Benchmarks
18. FAQ

---

Why Speculative Decoding Works (Bandwidth Bottleneck)

LLM inference per-token cost has two parts:

• Compute: ~2 × params × batch × tokens FLOPs
• Memory bandwidth: load all weights from HBM into compute units (~params × bytes_per_weight per forward pass)

For a single user generating one token at a time on a 70B BF16 model: 140 GB of weights must move from HBM → SMs each step. On H100 (3 TB/s HBM), that's a 47ms floor regardless of how few tokens you generate. The actual matmul takes <1ms. So generating 1 token vs 5 tokens in a single forward pass costs roughly the same wall time — the bandwidth dominates.

Speculative decoding exploits this: speculate K tokens with a cheap process, run the target on K+1 positions in one forward pass, accept the prefix that matches what the target would have generated. If acceptance rate is α, you generate ~(1+α·K) tokens per target forward pass instead of 1.

---

The Core Algorithm

Given: target model T, draft generator D, current sequence x[:t]
1. Sample K draft tokens: d[1..K] = D(x[:t])
2. Run T on x[:t] + d[:K] in a single forward pass
   → get target logits at positions t, t+1, ..., t+K
3. For each position i in 1..K:
   - Sample target token from T's distribution at position t+i-1
   - If target token == d[i]: accept and continue
   - Else: stop, replace d[i] with target's token, discard d[i+1..K]
4. Append accepted tokens + corrective token to x
5. Repeat

For greedy decoding, "sample target token" = argmax. For sampling (temperature > 0), use Leviathan et al. (2023) rejection sampling: accept d[i] with probability min(1, p_target(d[i]) / p_draft(d[i])); on rejection, sample from max(0, p_target − p_draft) normalized.

Both modes produce output mathematically identical in distribution to non-speculative decoding. No quality loss.

---

Draft Model Speculation

Use a small aligned model as the draft generator. Typical pairings:

Except for the DeepSeek row, the acceptance ranges are rule-of-thumb estimates for greedy decoding with a family-matched draft, not published measurements; measure acceptance on your own prompts. DeepSeek V2 Lite is not a valid draft for V3 because the two use different tokenizers.

Requirements: same tokenizer, similar instruction-tuning style, smaller than target by 5-100x. Speedup at K=5 follows from the acceptance rate; the original paper (Leviathan et al.) reports 2-3x on T5-XXL, and sampling at higher temperature accepts fewer tokens than greedy.

---

EAGLE-2 and EAGLE-3

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) trains an auxiliary feature-prediction layer on top of the target. Instead of a separate draft model, EAGLE predicts the target's hidden states for the next K positions and decodes tokens from those hidden states.

Speedups are the figures in each paper's abstract, measured by the authors at batch size 1; production engines usually see less.

EAGLE-2 introduced tree attention: instead of a linear chain of K speculative tokens, build a tree of multiple candidate continuations and verify all branches in one forward pass. Higher effective K with same memory budget.

Trained EAGLE heads are available on HuggingFace for major target models (Llama 3.1, Qwen 2.5, DeepSeek). For custom targets, training EAGLE on 4 GPUs takes 4-8 hours.

---

Medusa Heads

Medusa (Cai et al., 2024) trains K parallel prediction heads on top of the target model. At inference, all K heads predict simultaneously — no separate draft pass needed.

Trade-offs vs EAGLE:

• Simpler architecture (K linear heads vs full draft layer)
• Lower acceptance rate than EAGLE in the EAGLE papers' comparisons
• Tree attention over the heads' candidates, but no feature-level drafting
• Speedup: over 2.2x for Medusa-1 and 2.3-3.6x for Medusa-2, per the Medusa paper

Medusa is being superseded by EAGLE-2/3 in production. For new deployments in 2026, prefer EAGLE-2 unless you specifically want the simpler training pipeline.

---

n-gram / Prompt-Lookup Decoding

Zero-training speculation: maintain a buffer of recent context tokens. When generating, look up the most recent N tokens in the prompt + generated text — if a match exists, copy the next K tokens from that match as the draft.

Prompt: "def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)"
User: "Now write iterative version."
Generation: "def fibonacci(n):\n    if n <= 1:\n        return n\n    [draft from prompt: a, b = 0, 1...]"

Acceptance depends almost entirely on how much the output repeats the context: it is high for edit operations, code refactors and summaries that quote the source, and low for open-ended chat. No paper above publishes a general figure for it, so measure on your own traffic.

Cost: zero extra VRAM, zero training. For input-grounded workloads (code edits, RAG answers, summarization), this is the first speculation method to try.

---

Multi-Token Prediction (MTP)

Some models (DeepSeek V3, Llama-Nemotron-Ultra, Hunyuan-Large) include MTP weights — auxiliary heads trained to predict the next 1-2 tokens beyond the current. At inference, MTP heads serve as built-in speculators.

# vLLM auto-detects MTP weights
vllm serve deepseek-ai/DeepSeek-V3 \
    --tensor-parallel-size 8 \
    --speculative-config '{"method": "deepseek_mtp", "num_speculative_tokens": 1}'

DeepSeek's V3 technical report says the second predicted token is accepted 85-90% of the time, which gives about 1.8x tokens per second, with no extra training or separate draft model to download.

---

vLLM Configuration

Draft Model

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-config '{"model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 5}'

EAGLE-2

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-config '{"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3-Instruct-70B", "num_speculative_tokens": 5}'

n-gram

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4, "prompt_lookup_min": 2}'

MTP (DeepSeek V3)

vllm serve deepseek-ai/DeepSeek-V3 \
    --tensor-parallel-size 8 \
    --speculative-config '{"method": "deepseek_mtp", "num_speculative_tokens": 1}'

See vLLM Complete Setup Guide.

---

SGLang Configuration

Checked against SGLang v0.5.23 (PyPI, October 30, 2026) and the SGLang speculative decoding docs on November 2, 2026. Current SGLang needs CUDA 13 and Python 3.11+.

# Install (per docs.sglang.io)
uv pip install --prerelease=allow sglang

# EAGLE-3 on Llama 3.3 70B
sglang serve \
    --model-path meta-llama/Llama-3.3-70B-Instruct \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path yuhuili/EAGLE3-LLaMA3.3-Instruct-70B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --tp 4

sglang serve is the command the current SGLang quickstart uses; python3 -m sglang.launch_server takes the same flags and is still what the speculative decoding page itself shows, so either works. The three speculative values above are the ones from SGLang's own EAGLE-3 example; raise --speculative-eagle-topk and --speculative-num-draft-tokens for a wider draft tree at batch 1, lower them under load. The same docs page covers EAGLE-2 (EAGLE), MTP for models that ship MTP heads, a separate small draft model (STANDALONE), and n-gram speculation (NGRAM).

---

TensorRT-LLM Configuration

For maximum H100/H200 throughput, build TRT-LLM engines with speculation:

trtllm-build \
    --checkpoint_dir /trt_ckpt/llama-70b \
    --output_dir /trt_engines/llama-70b-spec \
    --gemm_plugin bf16 \
    --speculative_decoding_mode eagle \
    --max_draft_len 5 \
    --max_input_len 4096 \
    --max_output_len 2048

Then serve via Triton with the EAGLE draft engine alongside the target. See TensorRT-LLM Setup.

---

llama.cpp Speculative Mode

./llama-speculative \
    -m models/llama-3.1-70b-Q5_K_M.gguf \
    -md models/llama-3.2-1b-Q8_0.gguf \
    -ngl 999 -ngld 999 \
    --draft 5 \
    -c 8192 \
    -p "Explain quantum entanglement to a high schooler."

-md is the draft model, --draft is K. At the end of a run llama-speculative prints how many drafted tokens were accepted; that acceptance rate, not a headline multiplier, tells you whether the pairing pays off on your prompts.

---

Tuning Acceptance Rate

Measure acceptance per workload. vLLM periodically logs a speculative-decoding metrics line (unless you start it with --disable-log-stats) with fields along these lines; the values are placeholders:

SpecDecoding metrics: Mean acceptance length: <L>, Accepted: <a> tokens, Drafted: <d> tokens,
  Per-position acceptance rate: <p1>, <p2>, ..., Avg Draft acceptance rate: <r>%

Tuning levers:

• num_speculative_tokens: 3-8. Sweet spot is usually 4-5. Higher K = more wasted compute on rejected drafts.
• Draft model size: bigger draft = higher acceptance, slower draft step. 1-3B for 70B target is standard.
• Temperature alignment: ensure draft and target use same sampling parameters.
• Tokenizer match: must be exact. Llama draft for Qwen target won't work.

Target: aim for acceptance >0.6 for net speedup at K=5. If acceptance <0.4, lower K or switch to n-gram.

---

Sampling vs Greedy: Speedup Trade-off

Rule-of-thumb ranges for a well-matched draft, not measurements; your acceptance rate is the number that matters.

For high-temperature creative generation, speculative decoding may not help. For chat (temp 0.5-0.7) and code (temp 0.0-0.3), it's a clear win.

---

Batched Serving Caveats

Speedup as a function of batch size:

Rule-of-thumb ranges, not measurements. The one published batched figure: EAGLE-3 reports 1.38x throughput at batch 64 in SGLang.

For high-throughput batch serving (LLM-as-a-service): focus on continuous batching, FP8, and PagedAttention rather than speculation. For interactive APIs (chatbots, copilots, agents): speculation is essential. See vLLM Complete Setup.

---

When Speculation Hurts

---

Decision Tree by Workload

Single-user chat / agent (batch=1):

1. Try n-gram speculation first (zero training, zero VRAM cost)
2. If acceptance >50%, you're done
3. If acceptance <50%, add EAGLE-2 (1-3 GB VRAM; its paper reports 3.05-4.26x)
4. If EAGLE not available for your model, use draft model (1-3B same family)

Code generation:

1. n-gram is excellent when the output reuses the context (edits, refactors); measure acceptance before stacking anything else
2. Combine with EAGLE for stacking gains

Long-context summarization / edit operations:

1. n-gram dominates, because much of the output copies the prompt
2. No need for separate draft

Batched API serving (batch>16):

1. Skip speculation
2. Focus on FP8, continuous batching, PagedAttention

MoE models (DeepSeek V3, Hunyuan-Large):

1. Use built-in MTP if available (zero training)
2. Otherwise EAGLE-2 with tree attention works well

---

Published Benchmarks

We have not run these methods on data-center GPUs ourselves, so this section collects the speedups the method authors published. Each comes from the paper's own setup (model, hardware, batch size), so compare the methods with care and measure acceptance on your workload.

Built-in multi-token prediction:

---

FAQ

See answers to common speculative decoding questions below.

---

Sources: Leviathan et al. (2023) Fast Inference from Transformers via Speculative Decoding | Cai et al. (2024) Medusa | Li et al. (2024) EAGLE-2 | Li et al. (2025) EAGLE-3 | vLLM speculative decoding docs | SGLang speculative decoding docs | Li et al. (2024) EAGLE | DeepSeek-V3 Technical Report. Papers read November 2, 2026.

Related guides:

• vLLM Complete Setup Guide
• TensorRT-LLM Setup
• CUDA Optimization
• Quantization Explained
• Mamba State-Space Models