PagedAttention & KV Cache Fragmentation
Compare standard contiguous KV memory allocation (which wastes 60-80% VRAM to internal & external fragmentation) against vLLM PagedAttention virtual block tables.
Serving Engine Parameters
In real production, requests rarely utilize the full allocated buffer, causing severe internal fragmentation in static engines.
KV Cache Allocation Architecture
Must pre-allocate contiguous slot for max_model_len regardless of actual user prompt size.
Allocates 16-token physical blocks on-demand via virtual page tables, enabling cross-request prefix caching (e.g. system prompts).
By eliminating external fragmentation and restricting internal fragmentation to the final block of a sequence (averaging block_size / 2 tokens per request), PagedAttention boosts GPU batch size by 2x to 5x with zero loss of FP16 numerical accuracy.