Speculative Decoding & Medusa: 2.5x Inference Speedups Without Quality Loss

1. The Memory Bandwidth Bottleneck (Memory Wall)

During the autoregressive decoding phase of an LLM, generating one token requires loading every single model parameter weight from High Bandwidth Memory (HBM) into the GPU SRAM computation cores. For a 70B parameter model in 16-bit precision:

On an NVIDIA H100 SXM5 GPU with 3.35 TB/s memory bandwidth, the theoretical maximum single-batch decoding throughput is strictly capped:

While the GPU compute cores (TFLOPS) are idling at less than 5% capacity during single-token forward passes, the memory bus is completely saturated. Speculative Decoding solves this by using a fast draft model to propose $K$ tokens in parallel, and then executing a single batched verification forward pass on the large model to validate all $K$ tokens at once.

2. Rejection Sampling Mathematical Proof: Lossless Verification

A common misconception among software engineers is that speculative decoding degrades model intelligence or reasoning precision. In reality, Leviathan et al. (2023) proved that modified rejection sampling guarantees the generated sequence follows the exact target distribution $p(x)$:

Because of this mathematical identity, speculative decoding produces output identical to calling the target model directly with seed-level reproducibility.

3. Medusa Heads vs. External Draft Models

4. Production vLLM & TensorRT-LLM Serving Configuration

Deploying speculative decoding in enterprise production requires configuring your inference engine with proper batch allocation and speculative tree widths:

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --enforce-eager

5. Interactive Speedup Calibrator

Test your own hardware configurations and acceptance rates using our dedicated interactive tool: