agent-readings

Reasoning, Acting, and Reflection

ReAct

ReAct prompts a frozen LLM to interleave natural-language thoughts with task actions and observations, making tool use and environment interaction part of the reasoning loop.

6Claims
6Evidence

Why It Matters

This paper gives a compact blueprint for the modern LLM agent loop: model proposes intent and action, host runtime executes the action, observation returns to context, and the trace remains inspectable.

Problem

  • Chain-of-thought style reasoning was useful but static: the model reasoned from internal representations without grounding in external state.
  • Language-model acting/planning work often predicted domain actions without abstract goal reasoning or working-memory style progress tracking.
  • Knowledge-intensive QA and fact verification needed both multi-hop reasoning and fresh retrieval, while interactive environments needed long-horizon subgoal tracking.

Key gap: The paper targets the missing closed-loop pattern that combines reasoning, action, and observation for general task solving rather than treating reasoning and acting as isolated capabilities.

Method

Augment the agent's action space with free-form language thoughts. Thoughts do not affect the external environment directly, but they update the context used to choose later thoughts or actions.

  • Provide a few ReAct-format trajectories as in-context examples.
  • Let the model generate thoughts and actions in an interleaved trajectory.
  • Execute actions through the environment or tool API, then append observations to context.
  • Use dense thoughts for knowledge tasks and sparse thoughts for longer-horizon decision tasks.

Claims

C1 · paper-supported

Adding language thoughts to the action space lets the model use reasoning as context for future actions without directly mutating the environment.

Scope: Formal setup and prompt-based ReAct method in Section 2.

  • E-method: The paper augments the action space with language thoughts that update context without environment feedback, then prompts a frozen LLM with human trajectories.

C2 · paper-supported

On HotpotQA and FEVER, ReAct is consistently better than acting-only prompting, while ReAct plus CoT self-consistency gives the strongest prompting results in Table 1.

Scope: PaLM-540B prompting on HotpotQA exact match and FEVER accuracy with a simple Wikipedia API.

  • E-hotpot-fever: ReAct beats Act on HotpotQA and FEVER (27.4 vs 25.7 EM; 60.9 vs 58.9 Acc). Hybrid methods reach 35.1 HotpotQA EM and 64.6 FEVER Acc.

C3 · paper-supported

External actions make ReAct trajectories more grounded than CoT, but the action structure introduces its own reasoning and retrieval failure modes.

Scope: Manual labeling of sampled HotpotQA trajectories.

  • E-failure-modes: ReAct has lower false positives than CoT in sampled successes (6% vs 14%) and no hallucination-labeled failures, but higher reasoning-error failures (47% vs 16%) and 23% search-result-error failures.

C4 · paper-supported

On ALFWorld, ReAct substantially improves over acting-only and imitation-learning baselines in success rate.

Scope: ALFWorld task-specific setup over 134 unseen evaluation games.

  • E-alfworld: Best-of-six ReAct reaches 71% ALFWorld success, compared with 45% for best-of-six Act and 37% for BUTLER.

C5 · paper-supported

On WebShop, one-shot ReAct improves success rate over Act and the IL/IL+RL baselines, but remains below human expert performance.

Scope: WebShop 500 test instructions with average score and success rate.

  • E-webshop: ReAct reaches WebShop score 66.6 and success rate 40.0, compared with Act 62.3/30.1, IL+RL 62.4/28.7, and human 82.1/59.6.

C6 · paper-supported

Flexible sparse reasoning beats an Inner-Monologue-like dense external feedback pattern on ALFWorld.

Scope: ReAct vs ReAct-IM ablation in ALFWorld Table 3.

  • E-react-im: Best-of-six ReAct reaches 71% ALFWorld success, while ReAct-IM reaches 53%; the paper attributes the gap to high-level goal decomposition and commonsense reasoning.

Agent Infrastructure Takeaways

  • The model should not directly execute tools; the host runtime should parse constrained actions and append observations.
  • Thoughts are useful as inspectable state, but they should not be treated as proof of correctness.
  • A review should record both positive evidence and failure modes, because retrieval and looping failures are part of the design contract.
  • Hybrid control is a real design knob: the paper's CoT-SC/ReAct switches show that internal reasoning and external retrieval can be routed based on confidence or timeout.

Caveats

  • ReAct is a prompting paradigm in this paper, not a new model architecture.
  • Standalone ReAct is not uniformly best: it trails CoT on HotpotQA exact match and needs hybrid routing for the strongest QA result.
  • Search quality is load-bearing; non-informative search accounts for a meaningful share of ReAct errors in the manual analysis.
  • The method can loop or repeat thoughts/actions under greedy decoding.
  • Decision-task results remain far below expert humans on WebShop.