agent-readings

Reasoning, Acting, and Reflection

Self-Refine

Self-Refine improves a model's first answer by repeatedly asking the same LLM to critique its output and rewrite it from that feedback.

6Claims
7Evidence

Why It Matters

For agents, Self-Refine turns intermediate artifacts into maintainable state: draft, feedback, revised draft, stop signal, and iteration budget are separate records that can be inspected, replayed, and connected to external checks.

Problem

  • Many LLM generations are useful but incomplete or suboptimal on the first attempt.
  • Prior iterative refinement methods often trained separate refiners or depended on supervised feedback data, scalar reward models, or human annotation.
  • One-shot prompting gives no durable place to store why an answer is wrong or how the next version should change.

Key gap: The paper asks whether a strong frozen LLM can improve its own outputs across tasks by generating task-specific natural-language feedback and then refining from that feedback.

Method

Generate an initial output, prompt the same LLM for specific feedback on that output, prompt the same LLM to refine the output using the feedback, append prior outputs and feedback to later prompts, and repeat until a stop condition or maximum iteration count.

  • Produce a first draft for the task.
  • Ask the model to critique the draft with specific, actionable feedback.
  • Ask the model to revise the draft using that feedback.
  • Carry the output and feedback history into the next iteration.
  • Return the final refinement after the stop condition or iteration budget is reached.

Claims

C1 · paper-supported

Self-Refine needs no supervised training data, additional training, or reinforcement learning; it uses the same LLM as generator, feedback provider, and refiner.

Scope: Abstract, Introduction, Section 2, and NeurIPS abstract page.

  • E-method: The paper defines Self-Refine as initial generation, self-feedback, and refinement using the same LLM, with no additional supervised training data, extra training, or reinforcement learning.

C2 · paper-supported

The loop retains previous outputs and feedback in the prompt, making intermediate drafts and critiques part of the computation state.

Scope: Section 2 algorithm and refinement equation discussion.

  • E-state-loop: The algorithm alternates feedback and refine steps until a stop condition. The paper says previous feedback and outputs are retained by appending them to the prompt, and the last refinement is returned.

C3 · paper-supported

Across seven evaluated tasks, Self-Refine improves strong base LLM outputs by about 20 absolute points on average over conventional one-step generation.

Scope: Abstract and Section 3 evaluation on GPT-3.5, ChatGPT, GPT-4, and code-model variants.

  • E-main-results: The abstract reports about 20 absolute points average improvement. Table 1 shows improvements across Sentiment Reversal, Dialogue Response, Code Optimization, Code Readability, Math Reasoning, Acronym Generation, and Constrained Generation for GPT-3.5, ChatGPT, and GPT-4.

C4 · paper-supported

The largest gains appear on tasks where feedback can point to missing constraints or preference defects, while math reasoning improves only modestly without stronger error signals.

Scope: Table 1 and Section 3.3 results discussion.

  • E-task-pattern: Examples include GPT-4 Dialogue Response improving from 25.4 to 74.6, ChatGPT Constrained Generation from 44.0 to 67.0, and GPT-4 Constrained Generation from 15.0 to 45.0. Math Reasoning changes are small in Table 1, such as GPT-4 92.9 to 93.1, and the paper attributes modest math gains to weak error identification.

C5 · paper-supported

Specific, actionable feedback is an important mechanism: generic or absent feedback reduces performance in the paper's ablations.

Scope: Section 4 feedback-quality analysis and Table 2.

  • E-feedback-quality: The feedback ablation reports lower scores with generic or no feedback. For ChatGPT Code Optimization, Self-Refine feedback scores 27.5 versus 26.0 generic feedback and 24.8 no feedback. For Sentiment Reversal, Self-Refine feedback scores 43.2 versus 31.2 generic feedback and 0 with no feedback.

C6 · paper-supported

Self-Refine depends on strong instruction-following models and has practical limits around iteration cost, non-monotonic refinement, English-only experiments, proprietary model dependence, and harmful-use risk.

Scope: Section 4 analysis and Section 6 limitations.

  • E-iterations-and-limits: The evaluation allows up to 4 iterations. Figure 4 reports average gains from more iterations with diminishing returns, while the analysis notes performance may not always increase monotonically. The Vicuna-13B experiment struggles to generate feedback in the required format or apply even oracle feedback.
  • E-formal-limits: The limitations section says Self-Refine requires sufficient few-shot or instruction-following ability, uses non-open-source models in the experiments, evaluates English datasets only, and does not explicitly guard against malicious prompting for harmful text.

Agent Infrastructure Takeaways

  • Treat generate, critique, refine, and stop-decision as separate logged runtime steps rather than one opaque completion.
  • Persist every intermediate output and feedback item so a review can explain why a slide, summary, or code artifact changed.
  • Do not treat self-feedback as a verifier. Where possible, pair critique prompts with external checks such as tests, parsers, graders, or task-specific constraints.
  • Iteration count is an infrastructure knob: it trades latency and cost for quality, and diminishing returns should be measured per task.
  • Refinement can be non-monotonic, so systems need selection policies, rollback to earlier candidates, or scoring across iterations.
  • Weak local models may fail at the feedback protocol itself; schema validation and prompt-format checks are part of the runtime contract.

Caveats

  • Self-Refine is a test-time prompting and orchestration method, not a new trained model.
  • The strongest experiments use proprietary models such as GPT-3.5, ChatGPT, GPT-4, and Codex.
  • Preference-based evaluations rely partly on humans and GPT-4-as-judge, so they are not all executable verifiers.
  • Math reasoning gains are modest without an external signal that identifies wrong answers.
  • More iterations increase inference cost and can produce non-monotonic quality changes.
  • The paper's evaluation is English-only and does not address safeguards against harmful prompted refinements.