🖥️ Orca Iteration-Level Batching • vLLM Scheduler • TensorRT-LLM • Zero Bubbles
Continuous Batching & GPU Utilization Profiler
Compare legacy static request-level batching with iteration-level continuous batching to unlock 3x higher serving throughput on NVIDIA GPUs.
Workload Profile
32 requests
25 req/sec
Serving Throughput
3.4x
Tokens per second boost
GPU Bubble Time
54% → 4%
Idle tensor core reduction
p99 Queue Delay
-72%
Immediate slot allocation
Batch Scheduling Architecture
1. Static Request-Level Batching Massive Idling "Bubble Time"
Fast Requests (Finished)
GPU Idle Bubbles (Wasted)
Slowest Stream
Short requests finish early, but their GPU slots remain completely idle until the longest request completes.
2. Iteration-Level Continuous Batching (vLLM / Orca) Continuous 100% Slot Filling
Dynamic Iteration Slot Reuse (Instant Replenishment upon EOS)
The scheduler checks for newly arrived requests at every single decode iteration step, backfilling slots instantly.
🚀 Synergy with PagedAttention:
Continuous batching is only viable because PagedAttention manages memory dynamically in non-contiguous virtual blocks. Without PagedAttention, dynamically injecting new requests into active GPU batches would cause catastrophic out-of-memory errors due to fragmented contiguous tensors.