Why It Matters
This paper turns inference-time sampling into a reliability primitive for agents: spend more model calls, aggregate final answers, and get a stronger signal than one greedy reasoning trace.
Problem
- Chain-of-thought prompting used greedy decoding in the original setup, producing one reasoning path and one final answer.
- Greedy decoding is locally optimal and repetitive, while a single sampled trace is stochastic and unreliable.
- Verifier or reranker approaches can improve generated reasoning, but add training, annotations, or auxiliary models.
Key gap: The paper asks whether a frozen model can improve reasoning accuracy by generating multiple diverse traces and marginalizing over their answers without additional training.
Method
Keep the same CoT prompt, but replace greedy decoding with sampling. Parse the final answer from each sampled reasoning path, discard the paths during aggregation, and output the answer with the highest vote or normalized weighted support.
- Prompt the language model using the same CoT exemplars as prior work.
- Sample a diverse set of reasoning paths from the decoder instead of taking the greedy path.
- Parse the final answer from each reasoning path.
- Aggregate answers by marginalizing out reasoning paths, usually via unweighted majority vote.
- Return the most consistent final answer.
Claims
C1 · paper-supported
Self-consistency is an unsupervised decoding method that needs no extra training, auxiliary verifier, or human annotation beyond the CoT prompt.
Scope: Method definition in Sections 1-2.
- E-method: Self-consistency prompts with CoT, samples diverse reasoning paths, parses their final answers, and chooses the most consistent answer; the paper describes it as off-the-shelf, unsupervised, and requiring no additional training, auxiliary models, finetuning, or human annotation.
C2 · paper-supported
For PaLM 540B, direct majority vote performs about as well as normalized probability-weighted aggregation and far better than greedy decoding on the evaluated tasks.
Scope: Aggregation comparison in Table 1.
- E-aggregation: For PaLM 540B, greedy decode gets 56.5 GSM8K, 94.7 MultiArith, 35.8 AQuA, 79.0 SVAMP, 79.0 CSQA, and 85.2 ARC-c; majority vote gets 74.4, 99.3, 48.3, 86.6, 80.7, and 88.7. Normalized weighted sum is nearly identical.
C3 · paper-supported
Self-consistency substantially improves arithmetic reasoning over greedy CoT across UL2, LaMDA, PaLM, and GPT-3/Codex model families.
Scope: Arithmetic benchmarks AddSub, MultiArith, ASDiv, AQuA, SVAMP, and GSM8K.
- E-arithmetic: Self-consistency improves every arithmetic row shown. Examples include PaLM 540B GSM8K 56.5 to 74.4 (+17.9), PaLM 540B AQuA 35.8 to 48.3 (+12.5), code-davinci-002 GSM8K 60.1 to 78.0 (+17.9), and code-davinci-002 AQuA 39.8 to 52.0 (+12.2).
C4 · paper-supported
The largest arithmetic gains appear on hard tasks such as GSM8K and AQuA, with PaLM 540B gaining +17.9 on GSM8K and +12.5 on AQuA, and code-davinci-002 gaining +17.9 on GSM8K and +12.2 on AQuA.
Scope: Table 2 arithmetic results.
- E-arithmetic: Self-consistency improves every arithmetic row shown. Examples include PaLM 540B GSM8K 56.5 to 74.4 (+17.9), PaLM 540B AQuA 35.8 to 48.3 (+12.5), code-davinci-002 GSM8K 60.1 to 78.0 (+17.9), and code-davinci-002 AQuA 39.8 to 52.0 (+12.2).
C5 · paper-supported
Self-consistency improves commonsense and symbolic tasks, including StrategyQA, ARC-challenge, last-letter concatenation, and coin-flip variants.
Scope: Commonsense and symbolic experiments in Table 3 and related discussion.
- E-commonsense-symbolic: The abstract reports improvements including StrategyQA +6.4 and ARC-challenge +3.9. The experiments also include symbolic last-letter concatenation and coin-flip tasks from the CoT paper.
C6 · paper-supported
Self-consistency is limited by parseable-answer assumptions and inference cost: it is most direct for fixed-answer tasks and requires many sampled generations.
Scope: Section 2 applicability note, Section 3.2 sample count, and limitations discussion.
- E-applicability: The paper states self-consistency can be applied only to problems where the final answer is from a fixed answer set, while open-text generation would need a consistency metric between generations.
- E-sampling-cost: Main results are averaged over 10 runs, sampling 40 outputs independently from the decoder in each run.
Agent Infrastructure Takeaways
- Self-consistency makes inference-time compute a reliability knob: run multiple traces, parse answers, and aggregate instead of trusting one trajectory.
- The method creates a clean separation between trace generation and answer aggregation; an agent runtime can store all sampled traces but act only on the aggregated answer.
- Majority voting works only after a task-specific answer parser, so structured output and answer normalization become part of the runtime contract.
- Sampling improves reliability but multiplies model calls; schedulers and serving stacks must budget for parallel or batched trace generation.
- For open-ended agent work, the missing piece is a consistency metric: exact-answer voting does not directly apply to free-form plans or side-effecting actions.
Caveats
- The method is a decoding strategy layered on CoT, not a new reasoning model.
- It assumes a fixed or parseable final answer; open-text generation needs a separate agreement metric.
- It increases inference cost substantially because the main setup samples 40 outputs per run.
- It can amplify biases shared across sampled traces if many wrong traces converge on the same wrong answer.
- The strongest results use large models and benchmark-style tasks with automatic answer checking.