agent-readings

Reasoning, Acting, and Reflection

Reflexion

Reflexion teaches an LLM agent across trials by turning task feedback into verbal self-reflections stored in memory, without updating model weights.

6Claims
6Evidence

Why It Matters

Reflexion makes agent learning a runtime data problem rather than only a training problem: trajectories, evaluator signals, reflections, and memory windows become first-class state.

Problem

  • LLM agents could act in environments using prompting, but mostly learned from in-context examples rather than from their own failed trials.
  • Traditional reinforcement learning can learn from reward, but often requires many samples and expensive model updates.
  • Scalar rewards provide weak credit assignment for long language-agent trajectories.

Key gap: The paper asks whether agents can improve over repeated attempts by converting feedback into natural-language experience summaries and reusing those summaries as memory.

Method

Run the agent for a task attempt, evaluate the trajectory, generate a natural-language reflection from the feedback and trajectory, append that reflection to memory, and retry with the memory in context until success or a trial limit.

  • Generate an initial trajectory with an Actor such as ReAct or CoT.
  • Evaluate the result with task-specific feedback.
  • Ask the Self-Reflection model to convert feedback and trajectory into a useful lesson.
  • Store the reflection in memory.
  • Retry the task with memory included in the Actor context until success, repeated failure, or max trials.

Claims

C1 · paper-supported

Reflexion reinforces an agent through verbal self-reflection stored in memory rather than through weight updates.

Scope: Framework definition in Sections 1 and 3.

  • E-method: The paper defines Reflexion as verbal reinforcement where an agent reflects on task feedback, stores reflective text in memory, and conditions future trials on that memory without updating LLM weights.

C2 · paper-supported

The framework decomposes learning into Actor, Evaluator, Self-Reflection, and memory components.

Scope: Section 3 modular formulation and Figure 2 algorithm.

  • E-components: Reflexion uses an Actor to generate text/actions, an Evaluator to score trajectories, a Self-Reflection model to generate verbal cues, and short/long-term memory. The paper bounds reflection memory, usually to 1-3 stored experiences.

C3 · paper-supported

On ALFWorld, ReAct plus Reflexion solves far more tasks than ReAct alone over repeated trials.

Scope: 134 ALFWorld environments across six task types.

  • E-alfworld: On 134 ALFWorld tasks, ReAct + Reflexion completes 130 of 134 tasks using the simple heuristic, and the paper summarizes an absolute 22% improvement over strong baselines in 12 iterative learning steps.

C4 · paper-supported

On HotPotQA, Reflexion improves both CoT-style reasoning and ReAct-style retrieval/reasoning over repeated attempts.

Scope: 100 HotPotQA questions with exact-match feedback and memory size 3.

  • E-hotpotqa: The paper reports Reflexion outperforms CoT-only, ReAct-only, and CoT(GT)-only baselines across several learning steps on 100 HotPotQA questions, and summarizes a 20% reasoning improvement.

C5 · paper-supported

On coding tasks, Reflexion achieves the paper's headline 91% pass@1 on HumanEval and improves several Python/Rust code-generation benchmarks.

Scope: HumanEval, MBPP, HumanEval Rust, and LeetcodeHardGym experiments.

  • E-coding: The abstract reports 91% pass@1 on HumanEval, surpassing the previous GPT-4 80% baseline. Section 4.3 says Reflexion outperforms all baseline accuracies and sets new state-of-the-art standards on all Python/Rust benchmarks except MBPP Python.

C6 · paper-supported

Reflexion can fail when feedback or self-tests are misleading, when memory is too small, or when verbal policy optimization gets stuck in a local minimum.

Scope: Coding analysis and limitations section.

  • E-limitations: The paper analyzes false positives/false negatives from generated unit tests, reports HumanEval Rust ablations where Reflexion with tests and self-reflection reaches 0.68 versus 0.60 baseline, and states Reflexion may get stuck in local minima while memory is limited to a sliding window.

Agent Infrastructure Takeaways

  • Reflexion makes failed trajectories reusable data. A runtime should persist trajectory, evaluator result, reflection text, and retry attempt together.
  • Feedback amplification is a separate component: scalar reward or binary failure is not enough unless converted into an actionable memory item.
  • Reflection memory needs lifecycle rules: size limits, replacement, retrieval, and stale or misleading lessons become correctness concerns.
  • External evaluators matter. Exact match, environment completion, compiler logs, and unit tests all create different reflection quality and failure modes.
  • For coding agents, generated tests are part of the trust boundary; false positives can make the agent accept wrong code.

Caveats

  • Reflexion is not gradient-based reinforcement learning and does not update model weights.
  • The method depends on the LLM's self-evaluation or on task heuristics; poor feedback can produce poor reflections.
  • The memory window is small in the paper's experiments, usually 1-3 reflections, and can lose useful history.
  • Repeated retries increase cost and can still converge to non-optimal local behavior.
  • Code-generation success is bounded by the quality and coverage of generated tests or compiler feedback.