agent-readings

Evaluation and Benchmarks

ClawBench

ClawBench evaluates browser agents on 153 write-heavy everyday tasks across 144 live production websites while intercepting the terminal request and retaining five synchronized evidence layers for scoring.

7Claims
7Evidence

Why It Matters

It treats browser-agent evaluation as an infrastructure problem spanning isolation, effect control, recording, reference collection, and outcome verification rather than as action prediction over static pages.

Problem

  • Static-trace web benchmarks evaluate action prediction over recorded pages rather than end-to-end task execution.
  • Self-hosted web environments improve reproducibility but omit production-site drift, authentication flows, anti-automation systems, and platform-specific validation.
  • Earlier live-web benchmarks primarily emphasize read-only information retrieval instead of purchases, reservations, applications, and detailed form submissions.
  • Executing write-heavy tasks on production services risks creating orders, messages, reservations, or other irreversible side effects.
  • URL, final-page, screenshot, or reference-action matching can miss alternative valid paths and does not expose enough evidence for auditing consequential workflows.

Key gap: The missing capability is a safe, auditable way to measure whether an agent can complete heterogeneous, state-changing daily web workflows on production websites.

Method

Collect broad everyday write-heavy tasks, have a human complete each task to identify the terminal committing request, run every model-task pair in a fresh instrumented browser container, block the matching terminal request at runtime, and score the retained agent trace against the human reference with a fixed Agent-as-Judge protocol.

  • Gather realistic write-heavy tasks from everyday website use and filter out read-only, unavailable, geographically restricted, paid, hard-verification, or unbounded-side-effect workflows.
  • Formalize each retained task as a task card with required personal data, expected final state, and a terminal submission target.
  • Record a human reference run and annotate the terminal request's URL pattern, method, and required payload fields.
  • Launch each model-task pair in a fresh isolated container with a clean Chromium profile, fixed resources, shared task data, and common browser actions.
  • Let the agent interact with the live production website while the extension and CDP sidecar record browser and network activity.
  • Capture and block the annotated terminal request before it reaches the server; leave ordinary page, image, analytics, and other non-terminal traffic intact.
  • Assemble the five-layer trace and compare it with the human reference and terminal payload through the fixed Agent-as-Judge prompt.
  • Aggregate binary verdicts into overall and category-level success rates, then use the recorded traces for failure diagnosis.

Claims

C1 · paper-supported

The paper's benchmark contains 153 everyday tasks spanning 144 live production websites and 15 fine-grained life and work categories.

Scope: Figure 1, Table 1, and Section 2.1.

  • E-scale: The paper reports 153 tasks across 144 live production websites, organized into 8 high-level groups and 15 fine-grained categories.

C2 · paper-supported

ClawBench evaluates write-heavy, state-changing workflows on live production websites rather than action prediction on static traces or interaction only in self-hosted replicas.

Scope: Introduction, Table 1, and Sections 2 and 2.1.

  • E-live-write-heavy: Tasks involve purchases, reservations, applications, account updates, and detailed form submissions on production websites, carrying user-specific data through platform validation toward a state-changing action.

C3 · paper-supported

The final-request interceptor captured and blocked every annotated terminal request reached in the 153 human reference runs, with zero false-positive blocks on recorded navigation traffic.

Scope: Section 2.2 and Appendix B.1.

  • E-interception-audit: The interceptor blocked every annotated terminal request reached in the 153 human reference runs and produced zero false-positive blocks on recorded navigation traffic.

C4 · paper-supported

Every run is recorded as five synchronized evidence layers and judged against a human reference trajectory with an outcome-oriented binary protocol.

Scope: Section 2.3 and Figure 4.

  • E-trace-judge: The five-layer trace is session video, action screenshots, HTTP traffic, agent messages, and browser actions; Agent-as-Judge compares it with the human reference and intercepted payload to return a binary verdict and justification.

C5 · paper-supported

On the reported human-adjudicated comparisons, Agent-as-Judge agreement is 93.46% for Claude Sonnet 4.6 trajectories and 84.97% for GPT-5.4 trajectories.

Scope: Table 2 and the Limitations section.

  • E-judge-agreement: Raw agreement with human verdicts is 93.46% for Claude Sonnet 4.6 and 84.97% for GPT-5.4 over the reported 153-task comparisons.

C6 · paper-supported

Across the shared eight-model evaluation, the highest overall success rate is 33.3%, and 68 of 153 tasks are solved by none of the evaluated models.

Scope: Table 3 and Section 3.2.

  • E-main-results: Claude Sonnet 4.6 scores 33.3%, Qwen 3.5 scores 26.1%, and GLM-5 scores 24.2%; 68 of 153 tasks are solved by no model and no task is solved by all eight.

C7 · paper-supported

Each model-task pair runs in a fresh isolated container with a clean Chromium profile, fixed resource limits, a 30-minute wall-clock cap, and a five-minute idle watchdog.

Scope: Section 3.1 and Appendix E.

  • E-run-isolation: Each run uses an isolated container and fresh Chromium profile with 16 GB RAM, 8 vCPU, 32 GB disk, a 1,800-second wall-clock limit, and a five-minute idle timeout.

Agent Infrastructure Takeaways

  • A browser-agent benchmark measures a model-plus-harness system; the harness, prompt, browser configuration, and action interface must be held constant for comparative claims.
  • Effect control is strongest at a narrow, machine-checkable boundary: the terminal network request is easier to audit and block than an agent's stated intention.
  • The safety envelope is task-scoped. Workflows with multiple irreversible intermediate effects should be excluded rather than treated as covered.
  • Auditability requires synchronized evidence from the UI, network, agent transcript, low-level actions, and full-session replay.
  • Human references remain useful when they specify the target state and field bindings without forcing the agent to copy one exact action sequence.
  • Fresh browser profiles and isolated containers reduce cross-run leakage, but production-site and provider drift still need to be recorded and reported.
  • Binary success should be paired with trace-level diagnostics so last-mile hesitation, anti-bot blocking, early exit, and malformed submissions remain visible.

Caveats

  • Live production websites trade rerun determinism for ecological validity; layouts, A/B tests, region, account state, and anti-automation systems can change.
  • Final-request interception is not a general browser sandbox and only covers the audited terminal endpoint for each task.
  • Live sessions, model calls, network instrumentation, and trace-based judging make repeated trials and ablations more expensive than static benchmarks.
  • Reported model scores are conditioned on the shared OpenClaw harness and should not be read as model-only capability measurements.
  • Agent-as-Judge uses a binary pass/fail metric that compresses intermediate progress and retains non-zero disagreement with human reviewers.
  • Claude Sonnet 4.6 shares a model family with the judge, creating a potential evaluator-family effect despite the reported human audit.
  • The breadth-first suite underrepresents mobile-only, non-English, and accessibility-dependent workflows.
  • The paper's model results are a dated snapshot; later models, harnesses, site states, or benchmark versions can change the ranking.