agent-readings · Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Why It Matters

CoT is the precondition for many agent loops: it turns an LLM call from answer selection into an inspectable intermediate-state update that later systems reuse for planning, tool use, critique, and search.

Problem

Large language models scaled well on many NLP tasks but still struggled with arithmetic, commonsense, and symbolic reasoning.
Rationale-augmented training and finetuning could teach intermediate steps, but required many high-quality rationales.
Standard few-shot prompting used input-output examples and often failed to improve substantially on reasoning tasks with scale alone.

Key gap: The paper tests whether a small number of natural-language reasoning exemplars can elicit multi-step reasoning from off-the-shelf LMs.

Method

Replace ordinary input-output few-shot exemplars with input-rationale-output triples. At test time, let the model generate intermediate reasoning text before the answer, allocating more inference-time tokens to problems that need multi-step reasoning.

Manually compose a small set of chain-of-thought exemplars for each task family.
Prompt off-the-shelf LMs with those exemplars instead of finetuning.
Decode model completions and evaluate only the final answer against benchmark labels.
Compare against standard few-shot input-output prompting across arithmetic, commonsense, and symbolic tasks.
Run ablations that remove natural-language reasoning or move it after the answer.

Claims

C1 · paper-supported

CoT prompting requires only few-shot exemplars with intermediate reasoning steps and does not finetune the model.

Scope: Method definition and reproducibility statement.

E-method: The paper defines CoT prompting as few-shot exemplars consisting of input, chain of thought, and output, with eight manually composed math exemplars used across most math benchmarks and four AQuA exemplars from training data.
E-reproducibility: The authors state that all experiments use prompting-based inference only, no finetuning was done, exact prompts are provided, GPT-3 API experiments are included, and PaLM 540B inference used TPU v4 4x4x12 with 192 chips / 384 cores.

C2 · paper-supported

On arithmetic reasoning, CoT improves large-model performance substantially and lets PaLM 540B set new state of the art on GSM8K, SVAMP, and MAWPS in the paper's comparison.

Scope: Five math word-problem benchmarks using GPT-3, LaMDA, PaLM, UL2, and Codex models.

E-arithmetic-results: CoT more than doubled GSM8K performance for the largest GPT and PaLM models; PaLM 540B with CoT achieved new state of the art on GSM8K, SVAMP, and MAWPS, while AQuA and ASDiv were within 2% of prior state of the art.

C3 · paper-supported

The paper finds CoT to be an emergent ability of scale: it helps around 100B+ parameter models but not smaller ones.

Scope: Arithmetic and symbolic scaling curves, especially PaLM/LaMDA/GPT model families.

E-scale-emergence: The authors report that CoT does not positively affect smaller models and only yields performance gains for models of roughly 100B parameters; smaller models produced fluent but illogical chains of thought.
E-symbolic-results: On last-letter concatenation and coin-flip toy tasks, PaLM 540B with CoT reaches almost 100% in-domain solve rates and shows upward scaling curves on longer out-of-domain sequences, while standard prompting fails on OOD variants.

C4 · paper-supported

CoT also improves several commonsense reasoning tasks, including StrategyQA and sports understanding for PaLM 540B.

Scope: CSQA, StrategyQA, Date Understanding, Sports Understanding, and SayCan evaluations.

E-commonsense-results: For PaLM 540B, CoT improves commonsense evaluations and reaches 75.6% on StrategyQA versus a 69.4% prior supervised best, and 95.4% on Sports Understanding versus an 84% unaided sports enthusiast baseline; gain on CSQA is described as minimal.

C5 · paper-supported

Natural-language intermediate reasoning matters: equation-only, variable-compute-only, and reasoning-after-answer ablations do not match full CoT on GSM8K.

Scope: Ablation study using LaMDA 137B and PaLM 540B, with additional datasets in appendix tables.

E-ablation: Equation-only prompting does not help much for GSM8K, variable-compute-only performs about the same as baseline, and placing reasoning after the answer also performs about the same as baseline.

C6 · paper-supported

CoT traces are useful for inspection but are not guaranteed to be faithful or correct reasoning paths, and serving them is costly because the effect appears at large model scales.

Scope: Manual error analysis and discussion limitations.

E-error-analysis: Among 50 correct LaMDA 137B GSM8K examples, all generated chains were logically and mathematically correct except two coincidental correct answers; among 50 wrong answers, 46% were almost correct with minor mistakes and 54% had major semantic or coherence errors.
E-limitations: The paper states that CoT does not settle whether the model is actually reasoning, has no guarantee of correct reasoning paths, can produce wrong paths, and is costly in real applications because it emerges only at large model scales.

Agent Infrastructure Takeaways

CoT turns reasoning into explicit per-call runtime state: the trace becomes something downstream agents can inspect, critique, route, or continue.
Inference-time compute is a control knob. Harder tasks can spend more tokens on intermediate reasoning, but this directly affects latency and serving cost.
Trace text is not a correctness proof. Agent runtimes should pair reasoning traces with executable checks, tool observations, or external verifiers.
Prompted reasoning quality depends on model scale and exemplar design, so production systems should treat CoT as a capability with evaluations rather than a guaranteed behavior.
CoT creates the interface later methods extend: self-consistency samples many traces, ReAct interleaves traces with actions, and reflection methods critique traces after failure.

Caveats

The paper evaluates prompting, not a trained reasoning architecture.
The strongest results use very large models, including PaLM 540B, which makes cost and access load-bearing.
The paper uses manually written exemplar rationales and does not eliminate prompt construction effort.
Reasoning traces can be wrong or coincidentally lead to correct answers.
Arithmetic and symbolic gains are strongest on tasks where answers are easy to grade; open-ended agent tasks still need external validation.