Why It Matters
Agents create many short and long generations with unpredictable lengths. Orca is the first paper in this list that makes inference serving feel like a scheduler problem rather than only a model-kernel problem.
Problem
- Existing inference servers such as Triton/TensorFlow Serving were designed around request-level batching.
- Autoregressive Transformer generation runs one model iteration per output token.
- Requests in a batch can need different output lengths, so short requests wait for longer ones and new arrivals wait for the whole batch to finish.
- Naively changing the batch after each token breaks simple batched execution because attention tensors have variable shapes across sequence lengths.
Key gap: The paper asks how to keep batching efficient while allowing the serving system to make scheduling decisions after every generation iteration.
Method
Run one token iteration at a time, update request membership after each iteration, selectively batch parameter-heavy operations while handling attention's variable-length state separately, and coordinate model-parallel workers with memory-aware scheduling.
- Admit an initial batch of requests.
- Invoke the model execution engine for one iteration rather than the whole request.
- After the iteration returns, immediately return finished requests to clients.
- Admit newly arrived requests into free batch slots before the next iteration.
- Apply batching to parameter-heavy operations such as linear layers and MLPs.
- Handle attention separately where per-request tensor shapes differ.
- Track KV memory growth so the scheduler does not admit more requests than memory can support.
- Use intra-layer/inter-layer model parallelism and pipelining for large models.
Claims
C1 · paper-supported
Autoregressive generation breaks request-level batching because each request can require a different number of model iterations.
Scope: Abstract and Section 1.
- E-request-batching-problem: The paper explains that existing request-level serving systems cannot change a batch until all requests in the batch finish, so short requests wait for long ones and new arrivals wait for the current batch.
C2 · paper-supported
Iteration-level scheduling lets newly arrived requests wait only for one model iteration and lets completed requests return immediately after their final token.
Scope: Abstract and Section 1.
- E-iteration-scheduling: Orca invokes the engine to run one model iteration on the batch, then checks completion and admits new requests after every iteration.
C3 · paper-supported
Selective batching batches only selected operations and handles attention differently because attention tensors have variable shapes as sequence lengths diverge.
Scope: Abstract and Section 1.
- E-selective-batching: Selective batching applies batching only to selected operations, while attention is processed per request because its tensor shapes depend on already processed tokens and it has no model parameters to amortize.
C4 · paper-supported
Orca includes distributed serving support for models with hundreds of billions of parameters through intra-layer and inter-layer model parallelism.
Scope: Section 1 and system design sections.
- E-distributed-serving: The paper states Orca adopts intra-layer and inter-layer model parallelism and adds scheduling, memory management, and pipelined execution for large-scale models.
C5 · paper-supported
On GPT-3-style models up to 341B parameters, Orca reports up to 36.9x throughput improvement over FasterTransformer at the same latency level.
Scope: Abstract and Section 1.
- E-throughput: The evaluation on GPT-3 175B and additional GPT-3-style models up to 341B reports 36.9x throughput improvement at the same latency level compared with NVIDIA FasterTransformer.
C6 · paper-supported
Orca still treats KV-cache memory as a central scheduling constraint because attention state grows with generated sequence length.
Scope: Sections 1-3 and memory design.
- E-kv-growth: The paper describes Transformer key/value state as growing with the number of processed tokens, requiring the scheduler to consider memory while requests continue generating.
Agent Infrastructure Takeaways
- LLM serving for agents is scheduling, not just kernels: every agent step consumes a variable number of token iterations.
- Iteration-level scheduling is the ancestor of modern continuous batching. Agents need it because task lengths are unpredictable.
- Short generations should not wait behind long generations from the same batch; the scheduler must retire completed sequences immediately.
- Admission control must account for KV-cache growth, not just current batch size.
- Serving optimizations change product behavior indirectly: lower per-step latency and higher throughput determine whether multi-call agent loops feel usable.
- Orca makes a useful separation between parameter-heavy operations worth batching and state-heavy attention operations that need different treatment.
Caveats
- The evaluation is on GPT-style language models, not every generative model domain.
- The main comparison is against the evaluated FasterTransformer baseline from the paper era.
- The paper predates PagedAttention/vLLM and modern production serving engines.
- Selective batching handles variable-length attention, but does not fully solve KV-cache memory fragmentation and sharing.
- Serving throughput gains do not guarantee semantic reliability for agents; they make retries, self-consistency, search, and multi-step workflows cheaper.