agent-readings · SWE-bench

<div class="slide-frame slide-frame--center">
  <p class="deck-kicker">agent-readings / Evaluation and Benchmarks</p>
  <h1>SWE-bench</h1>
  <p class="deck-subtitle">Coding agents graded by real tests</p>

<div class="deck-meta-strip">
    <div class="deck-meta-pill"><strong>2023</strong><span>ICLR 2024; arXiv 2023</span></div>
    <div class="deck-meta-pill"><strong>7</strong><span>claims</span></div>
    <div class="deck-meta-pill"><strong>7</strong><span>evidence refs</span></div>
    <div class="deck-meta-pill"><strong>paper-read</strong><span>status</span></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">The one sentence</p>
  <h2>SWE-bench evaluates language models on real GitHub issues by asking them to patch repository snapshots and grading whether the repo's tests pass.</h2>
  <p class="deck-note">For agents, SWE-bench moved evaluation from generated snippets to repository-scale work with long context, tool execution, test feedback, and multi-file edits.</p>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Before this paper</p>
  <h2>The friction was structural.</h2>
  <div class="apple-card-grid apple-card-grid--three">
    <div class="apple-card"><p>Traditional coding benchmarks often use short self-contained problems that can be solved in a few lines.</p></div>
<div class="apple-card"><p>Modern coding assistants operate inside large repositories and must understand issue descriptions, dependencies, tests, and multi-file change sets.</p></div>
<div class="apple-card"><p>Language models were improving faster than evaluation settings that captured real software-engineering difficulty.</p></div>
  </div>
  <div class="deck-callout">
    <span>Key gap</span>
    <p>The paper asks how to build a realistic, challenging, and automatically verifiable benchmark for repository-scale software issue resolution.</p>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Core idea</p>
  <h2>Mine GitHub pull requests that resolve issues and add tests, validate them through execution, then evaluate models by applying generated patches and running the...</h2>
  <p class="deck-note">Mine GitHub pull requests that resolve issues and add tests, validate them through execution, then evaluate models by applying generated patches and running the associated tests.</p>
  <div class="idea-loop" aria-label="Agent loop">
    <div><strong>Model</strong><span>reason</span></div>
    <div><strong>Runtime</strong><span>act</span></div>
    <div><strong>World</strong><span>observe</span></div>
  </div>
  <div class="step-strip">
    <div class="step-item">
  <span>01</span>
  <p>Select popular Python repositories with substantial test suites.</p>
</div>
<div class="step-item">
  <span>02</span>
  <p>Scrape merged pull requests and issue links.</p>
</div>
<div class="step-item">
  <span>03</span>
  <p>Filter for PRs that resolve an issue and include test-file changes.</p>
</div>
<div class="step-item">
  <span>04</span>
  <p>Apply execution filtering: the test patch must create fail-to-pass tests, and the reference solution must make them pass.</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Action grammar</p>
  <h2>The agent is an interface contract.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>issue: user-submitted bug report or feature request.</p></div>
<div class="apple-card"><p>base commit: repository snapshot before the fixing pull request.</p></div>
<div class="apple-card"><p>reference PR: merged pull request associated with the issue.</p></div>
<div class="apple-card"><p>test patch: tests added by the reference PR.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Executable shape</p>
  <h2>The mechanism should fit on one screen.</h2>

```python
for pr in scraped_pull_requests:
    if not resolves_issue(pr) or not modifies_tests(pr):
        continue
    base = checkout(pr.base_commit)
    apply(test_patch(pr), base)
    before = run_tests(base)
    apply(solution_patch(pr), base)
    after = run_tests(base)
    if has_fail_to_pass(before, after) and installs_cleanly(base):
        add_instance(issue_text(pr), base_commit, test_patch(pr), tests)

for instance in benchmark:
    patch = model.generate_patch(instance.issue, instance.codebase)
    if apply_patch(patch, instance.codebase) and run_tests(instance.codebase):
        resolved += 1
```
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Evidence</p>
  <h2>The proof objects.</h2>
  <div class="proof-grid">
    <div class="proof-card">
  <span>E-dataset-size</span>
  <p>The paper introduces 2,294 task instances drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.</p>
</div>
<div class="proof-card">
  <span>E-evaluation-contract</span>
  <p>The model receives issue text and a codebase, generates a patch, and the patch is applied and evaluated by running unit/system tests; the metric is percentage resolved.</p>
</div>
<div class="proof-card">
  <span>E-construction</span>
  <p>The pipeline scrapes around 90,000 PRs, filters for issue-resolving PRs with test changes, and uses execution filtering requiring at least one fail-to-pass test and no...</p>
</div>
<div class="proof-card">
  <span>E-repository-scale</span>
  <p>The paper states tasks require understanding changes across functions, classes, and files; issue descriptions average 195 words, codebases contain thousands of files, and...</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Claim map</p>
  <h2>What the review actually supports.</h2>
  <div class="claim-grid">
    <div class="claim-card">
  <span>C1 · paper-supported</span>
  <p>SWE-bench contains 2,294 software-engineering problems from real GitHub issues and pull requests across 12 popular Python repositories.</p>
  <small>E-dataset-size</small>
</div>
<div class="claim-card">
  <span>C2 · paper-supported</span>
  <p>A model is evaluated by generating a patch for a repository snapshot; the patch must apply and pass the task's tests to count as resolved.</p>
  <small>E-evaluation-contract</small>
</div>
<div class="claim-card">
  <span>C3 · paper-supported</span>
  <p>Instances are mined through a three-stage pipeline: scraping PRs, filtering for issue-resolving PRs with tests, and execution filtering for fail-to-pass...</p>
  <small>E-construction</small>
</div>
<div class="claim-card">
  <span>C4 · paper-supported</span>
  <p>SWE-bench tasks require repository-scale reasoning over long inputs and often multi-function or multi-file edits.</p>
  <small>E-repository-scale</small>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Agent infrastructure</p>
  <h2>What changes if you build systems.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>Executable grading is the benchmark gold standard for coding agents: patches either apply and pass tests or they do not.</p></div>
<div class="apple-card"><p>SWE-bench forces the whole agent stack into view: retrieval, code navigation, editing, tool execution, test running, and retry policy.</p></div>
<div class="apple-card"><p>A benchmark harness is infrastructure. Environment setup, dependency pinning, patch application, and test selection determine what is actually...</p></div>
<div class="apple-card"><p>Repository-scale context is a systems problem, not just a model problem; agents must find the few relevant lines among thousands of files.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Caveats</p>
  <h2>What this does not prove.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>The original paper score is historical; current SWE-bench leaderboard numbers are much higher and should be checked live before citing SOTA.</p></div>
<div class="apple-card"><p>Tests are strong but not perfect specifications.</p></div>
<div class="apple-card"><p>The benchmark is Python-heavy and repository-selection biased.</p></div>
<div class="apple-card"><p>Generated patches can overfit to visible tests or exploit weak harnesses if the benchmark environment is not secured.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Run it</p>
  <h2>No PoC yet.</h2>
  <p class="deck-note">No PoC yet - contributions welcome. A small PoC could package one miniature repo issue with a failing test and evaluate a generated patch through the same apply-and-test contract.</p>
  <div class="reference-list">
    <ul><li>Paper: <a href="https://arxiv.org/abs/2310.06770">SWE-bench: Can Language Models Resolve Real-World GitHub Issues?</a></li><li>Project/code: <a href="https://www.swebench.com">https://www.swebench.com</a></li><li>hardens: <a href="https://arxiv.org/abs/2507.02825">Establishing Best Practices for Building Rigorous Agentic Benchmarks</a></li><li>broader_environment: <a href="https://arxiv.org/abs/2404.07972">OSWorld</a></li><li>earlier_agent_training: <a href="https://arxiv.org/abs/2112.09332">WebGPT</a></li></ul>
  </div>
</div>

Note: This deck is synthesized from `data/reviews/swe-bench-jimenez-2023.json`. Update the review record, then run `bun run build`.