Why It Matters
For agents, per-step latency compounds across tool loops, search, debate, retries, and self-consistency. Speculative decoding attacks the wall-clock cost of every generation without requiring a new target model.
Problem
- Generating K tokens from an autoregressive Transformer normally requires K serial target-model invocations.
- Large target-model decode steps are slow and often limited by memory bandwidth or communication rather than raw arithmetic.
- Adaptive computation methods can speed some cases but often require architecture changes, retraining, or accept changed output distributions.
- Serving systems need latency improvements that work with existing off-the-shelf models.
Key gap: The paper asks whether a small approximation model can propose tokens while a large target model verifies them in a way that preserves exact target-model sampling behavior.
Method
Generate a speculative token block with a cheap model, run the expensive target model once over that block to score all proposed positions, accept the longest prefix allowed by a distribution-preserving rule, and sample a corrected token when a draft token is rejected.
- Run the approximation model autoregressively for gamma draft tokens.
- Run the target model once on the prompt plus speculative tokens to obtain target probabilities for all draft positions.
- Walk through proposed tokens in order and accept each according to the speculative sampling rule.
- If all proposed tokens are accepted, sample one additional token from the target model's next-token distribution.
- If a proposed token is rejected, sample a correction from the residual distribution and stop the round.
- Append accepted and corrected tokens to the output.
- Repeat until termination.
- Choose approximation model and gamma to balance target-pass savings against rejected draft work.
Claims
C1 · paper-supported
Standard autoregressive decoding requires K serial target-model runs to generate K tokens.
Scope: Abstract and Introduction.
- E-serial-decoding: The paper states that decoding K tokens takes K serial runs of the large autoregressive target model.
C2 · paper-supported
Speculative decoding proposes several tokens with an efficient approximation model and verifies them using the target model in parallel.
Scope: Abstract and Introduction.
- E-speculative-method: The paper proposes sampling a speculative prefix from an efficient approximation model, then running the target model on those proposed tokens to verify them in parallel.
C3 · paper-supported
The algorithm preserves the target model's output distribution rather than merely approximating it.
Scope: Abstract, Introduction, and method sections.
- E-distribution-preserving: The authors state the speculative sampling method can generate several tokens concurrently without changing the target distribution, and Figure 1 illustrates fewer target runs with unchanged probability.
C4 · paper-supported
The method can accelerate off-the-shelf models without retraining or architecture changes.
Scope: Abstract.
- E-off-the-shelf: The abstract says the method accelerates existing off-the-shelf models without retraining or architecture changes.
C5 · paper-supported
On T5-XXL, the paper reports 2x-3x latency acceleration versus the standard T5X implementation with identical outputs.
Scope: Abstract and Introduction.
- E-t5xxl-speedup: The paper reports 2x-3x acceleration over the standard T5X implementation on T5-XXL with identical outputs.
C6 · paper-supported
The paper evaluates tasks/models including a 97M GPT-like model, 11B T5-XXL for translation and summarization, and a 137B LaMDA dialog model.
Scope: Introduction.
- E-evaluation-scope: The paper says it analyzes unconditional generation with a 97M GPT-like model, translation and summarization with 11B T5-XXL, and a dialog task with 137B LaMDA.
Agent Infrastructure Takeaways
- Speculative decoding is a latency primitive for agent loops: every model call can return faster without changing the requested target distribution.
- The draft model becomes serving infrastructure. It must be co-located, versioned, monitored, and chosen per target/workload.
- Exactness matters for agent reliability. Speculative decoding's appeal is preserving outputs, not merely making a cheaper model answer first.
- Speedup depends on acceptance rate. Hard reasoning, long-tail domains, or mismatched draft models can reduce gains.
- Speculative decoding complements continuous batching and PagedAttention: one attacks serial depth, the others attack concurrency and memory.
- For agent search, faster decoding lowers the cost of self-consistency, debate, reflection, and tree expansion.
Caveats
- The method needs a useful approximation model; a weak draft model can waste work.
- The target distribution is preserved only if the sampling rule and probabilities are implemented exactly.
- The headline 2x-3x speedup is workload/model/implementation-specific.
- Speculative decoding reduces latency but does not reduce the semantic number of agent decisions.
- Serving integration can be nontrivial because the draft and target models must share tokenization and efficient scheduling.
- The method may interact with quantization, caching, batching, and structured decoding constraints in engine-specific ways.