Why It Matters
For agents, serving cost is dominated by long, branching, repeated contexts. PagedAttention made high-throughput LLM serving practical by treating context memory as an OS memory-management problem.
Problem
- High-throughput LLM serving requires batching many concurrent requests.
- Each request's KV cache is large, grows dynamically with generated tokens, and shrinks when a request finishes.
- Contiguous allocation wastes GPU memory through fragmentation and over-reservation, limiting batch size.
- Parallel sampling, beam search, and shared prompts duplicate large KV regions even when prefixes are identical.
Key gap: The paper asks how to manage KV-cache memory dynamically so the serving system can keep GPUs full without wasting memory or duplicating shared prefixes.
Method
Split KV caches into fixed-size blocks, allocate physical blocks on demand, map logical sequence blocks to physical GPU blocks, and make the attention kernel follow those mappings while the serving engine handles sharing, allocation, and eviction.
- Partition each sequence's KV cache into logical token blocks.
- Allocate physical KV blocks only as needed instead of reserving a contiguous maximum-length cache.
- Maintain a block table for each sequence.
- Modify attention kernels to read KV tensors by following block-table entries.
- Share physical KV blocks for common prefixes or decoding branches.
- Use copy-on-write when a shared block needs modification by one branch.
- Integrate the allocator with continuous batching so memory pressure informs admission and scheduling.
- Evaluate vLLM against FasterTransformer and Orca-style serving baselines.
Claims
C1 · paper-supported
KV-cache memory is huge, dynamically sized, and can be significantly wasted by fragmentation and redundant duplication in existing serving systems.
Scope: Abstract.
- E-kv-cache-problem: The paper states that KV-cache memory for each request is huge, grows and shrinks dynamically, and can be wasted by fragmentation and duplicate copies when managed inefficiently.
C2 · paper-supported
PagedAttention borrows classical virtual-memory paging ideas by allowing non-contiguous KV-cache storage through block mappings.
Scope: Abstract.
- E-pagedattention: The paper proposes PagedAttention, an attention algorithm inspired by virtual memory and paging techniques from operating systems.
C3 · paper-supported
vLLM builds on PagedAttention to achieve near-zero KV-cache memory waste.
Scope: Abstract.
- E-vllm-memory: vLLM is described as achieving near-zero waste in KV-cache memory by building on PagedAttention.
C4 · paper-supported
vLLM supports flexible KV-cache sharing within and across requests, including cases such as parallel sampling and shared prompts.
Scope: Abstract.
- E-sharing: The paper says vLLM flexibly shares KV cache within and across requests to further reduce memory usage.
C5 · paper-supported
The evaluation reports 2-4x throughput improvement at the same latency level over systems such as FasterTransformer and Orca.
Scope: Abstract.
- E-throughput: The evaluation reports that vLLM improves throughput by 2-4x at the same latency level compared with state-of-the-art systems such as FasterTransformer and Orca.
C6 · paper-supported
The paper reports larger gains for longer sequences, larger models, and more complex decoding algorithms.
Scope: Abstract.
- E-long-sequence-gains: The abstract says the throughput improvement is more pronounced for longer sequences, larger models, and more complex decoding algorithms.
Agent Infrastructure Takeaways
- Agent serving needs memory virtualization because long prompts, tool traces, and branchy reasoning make KV cache the scarce resource.
- PagedAttention is the same OS move as virtual memory: stable logical context, flexible physical placement.
- Copy-on-write KV sharing is directly useful for self-consistency, Tree of Thoughts, retries, and parallel tool-plan candidates that share a prefix.
- Serving engines should expose memory pressure as a scheduler signal, not hide it below the request API.
- The cost of agent reasoning is often KV-cache residency time, not just generated-token count.
- vLLM made modern continuous batching usable at higher concurrency because memory management stopped wasting so much GPU RAM.
Caveats
- The review uses the arXiv paper text and public metadata; detailed per-table benchmark interpretation should still be rechecked before quoting secondary numbers.
- PagedAttention improves memory management; it does not reduce model arithmetic per token by itself.
- Block-table indirection and allocator details matter for actual kernel performance.
- The 2-4x throughput result is workload/baseline-specific.
- Prefix sharing only helps when prompts or branches actually share tokens.
- Later serving systems may combine PagedAttention with speculative decoding, quantization, prefix caching, and kernel-level improvements.