Speculative Decoding & High-Throughput Inference: Draft Models, Medusa & vLLM
LLM autoregressive token generation is bottlenecked by GPU memory bandwidth, not compute. Learn how speculative decoding, multi-head drafting (Medusa, Eagle), and vLLM continuous batching unlock up to 4.2x faster token streaming with zero quality loss.
Cursor AI & Claude 3.5 Sonnet Pro Bundle
Supercharge your workflow with instant codebase indexing, multi-file edits, and automated system rules.
1. Modern Speculative Decoding Paradigms Compared
There are three primary architectural approaches to speculative acceleration in production:
| Method | Mechanism | Additional VRAM Cost | Average Speedup |
|---|---|---|---|
| Draft Model (e.g. Llama-3-1B) | Independent small autoregressive model | ~2.5 GB (weights + KV cache) | 2.1x - 3.2x |
| Medusa Multi-Heads | Multiple decoding MLP heads on final layer | < 150 MB (zero extra model) | 2.4x - 3.6x |
| Eagle-2 (Tree Attention) | Feature-level autoregression with tree search | ~500 MB (1 transformer layer) | 3.5x - 4.5x |
2. Configuring Speculative Decoding in vLLM (Production CLI)
Deploying speculative decoding with vLLM requires zero code changes to your application. Pass the draft model and speculative length flags directly to the server:
# Launch Llama-3.3-70B with Llama-3.2-1B as Speculative Draft Model
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 5 \
--speculative-draft-tensor-parallel-size 1 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.92 \
--max-model-len 16384 \
--port 80003. Measuring and Optimizing Acceptance Rate (γ)
The speedup of speculative decoding is governed by the average acceptance rate γ and draft model execution latency:
- Structured JSON and SQL schema queries (boilerplate syntax is predictable)
- Code generation in TypeScript, Python, and Go (keywords and boilerplate match easily)
- Low temperature (temperature < 0.2) greedy sampling tasks
- High-entropy creative fiction writing (temperature > 0.9)
- Highly competitive mathematical deduction with branching decisions
- Mismatch between tokenizer vocabularies between draft and target models
Frequently Asked Questions
Can draft and target models have different tokenizers?
No. Standard speculative decoding requires the draft and target models to share the exact same vocabulary and tokenizer (e.g. Llama-3 128k tokenizer).
How many speculative tokens (--num-speculative-tokens) should I set?
The optimal default is 4 or 5 tokens. Setting it higher than 7 leads to diminishing returns as probability of later token rejection compounds exponentially.
Cursor AI & Claude 3.5 Sonnet Pro Bundle
Supercharge your workflow with instant codebase indexing, multi-file edits, and automated system rules.