agent-readings

Evaluation and Benchmarks

SWE-bench

SWE-bench evaluates language models on real GitHub issues by asking them to patch repository snapshots and grading whether the repo's tests pass.

7Claims
7Evidence

Why It Matters

For agents, SWE-bench moved evaluation from generated snippets to repository-scale work with long context, tool execution, test feedback, and multi-file edits.

Problem

  • Traditional coding benchmarks often use short self-contained problems that can be solved in a few lines.
  • Modern coding assistants operate inside large repositories and must understand issue descriptions, dependencies, tests, and multi-file change sets.
  • Language models were improving faster than evaluation settings that captured real software-engineering difficulty.
  • Agent evaluation needs executable graders rather than only human or LLM judges.

Key gap: The paper asks how to build a realistic, challenging, and automatically verifiable benchmark for repository-scale software issue resolution.

Method

Mine GitHub pull requests that resolve issues and add tests, validate them through execution, then evaluate models by applying generated patches and running the associated tests.

  • Select popular Python repositories with substantial test suites.
  • Scrape merged pull requests and issue links.
  • Filter for PRs that resolve an issue and include test-file changes.
  • Apply execution filtering: the test patch must create fail-to-pass tests, and the reference solution must make them pass.
  • Construct benchmark inputs from issue text and a codebase snapshot.
  • Ask a model or agent to generate a patch.
  • Apply the patch with Unix patch tooling.
  • Run the repository's relevant tests and count the instance resolved only if the patch applies and tests pass.

Claims

C1 · paper-supported

SWE-bench contains 2,294 software-engineering problems from real GitHub issues and pull requests across 12 popular Python repositories.

Scope: Abstract and Section 2.

  • E-dataset-size: The paper introduces 2,294 task instances drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.

C2 · paper-supported

A model is evaluated by generating a patch for a repository snapshot; the patch must apply and pass the task's tests to count as resolved.

Scope: Section 2.2.

  • E-evaluation-contract: The model receives issue text and a codebase, generates a patch, and the patch is applied and evaluated by running unit/system tests; the metric is percentage resolved.

C3 · paper-supported

Instances are mined through a three-stage pipeline: scraping PRs, filtering for issue-resolving PRs with tests, and execution filtering for fail-to-pass behavior.

Scope: Section 2.1.

  • E-construction: The pipeline scrapes around 90,000 PRs, filters for issue-resolving PRs with test changes, and uses execution filtering requiring at least one fail-to-pass test and no install/runtime errors.

C4 · paper-supported

SWE-bench tasks require repository-scale reasoning over long inputs and often multi-function or multi-file edits.

Scope: Abstract and Section 2.3.

  • E-repository-scale: The paper states tasks require understanding changes across functions, classes, and files; issue descriptions average 195 words, codebases contain thousands of files, and reference solutions average 1.7 files, 3.0 functions, and 32.8 changed lines.

C5 · paper-supported

In the original evaluation, the best-performing model, Claude 2 with BM25 retrieval, resolved only 1.96% of issues.

Scope: Abstract and Introduction.

  • E-claude2-result: The paper reports Claude 2 with BM25 retrieval as the best-performing evaluated system, resolving 1.96% of issues.

C6 · paper-supported

The paper releases SWE-bench-train, a 19,000-instance non-testing dataset from 37 repositories, and SWE-Llama models fine-tuned from CodeLlama.

Scope: Introduction.

  • E-train-data: The paper releases SWE-bench-train with 19,000 non-testing task instances from 37 repositories and two SWE-Llama fine-tuned models.

C7 · paper-supported

SWE-bench is designed to be continually updatable from new GitHub issues and PRs, helping evaluate tasks created after model training dates.

Scope: Section 2.3.

  • E-updatable: The authors state the collection process can be applied to new Python repositories and issues, supporting continual updates and evaluations after model training dates.

Agent Infrastructure Takeaways

  • Executable grading is the benchmark gold standard for coding agents: patches either apply and pass tests or they do not.
  • SWE-bench forces the whole agent stack into view: retrieval, code navigation, editing, tool execution, test running, and retry policy.
  • A benchmark harness is infrastructure. Environment setup, dependency pinning, patch application, and test selection determine what is actually measured.
  • Repository-scale context is a systems problem, not just a model problem; agents must find the few relevant lines among thousands of files.
  • The original 1.96% score is a reminder that real issue resolution was far beyond early chat-completion baselines.
  • SWE-bench also exposes benchmark-security concerns: because tests are executable, sandboxing and anti-cheating controls matter.

Caveats

  • The original paper score is historical; current SWE-bench leaderboard numbers are much higher and should be checked live before citing SOTA.
  • Tests are strong but not perfect specifications.
  • The benchmark is Python-heavy and repository-selection biased.
  • Generated patches can overfit to visible tests or exploit weak harnesses if the benchmark environment is not secured.
  • Long-context and retrieval setup materially affect results.
  • SWE-bench evaluates coding issue resolution, not general-purpose agent autonomy.