agent-readings

Evaluation and Benchmarks

tau-bench

tau-bench evaluates whether tool-using agents can converse with simulated users, follow domain policies, update databases correctly, and do so consistently across repeated trials.

7Claims
7Evidence

Why It Matters

For production agents, pass@1 is not enough. tau-bench makes reliability visible by asking whether the same task succeeds repeatedly under stochastic user conversations.

Problem

  • Many tool-use benchmarks provide all required user information in a single instruction.
  • Function-calling benchmarks often evaluate individual tool calls rather than long-horizon dialogue plus database mutation.
  • Agent benchmarks often omit domain-specific policies that constrain what actions are allowed.
  • Single-run scores hide inconsistency when agents face stochastic users or slightly different conversation flows.

Key gap: The paper asks how to evaluate agents in realistic tool-agent-user interactions where user communication, API actions, and policy compliance all matter.

Method

Build domains with realistic databases, APIs, policy documents, and user scenarios; run agents in conversations with LM-simulated users; then compare final database state to annotated goal state across multiple trials.

  • Manually design domain schemas, APIs, and policy documents.
  • Generate realistic database entries with LM assistance and manual verification.
  • Create and verify user scenarios and target goal states.
  • Run an agent with access to policy text and API tools.
  • Let a simulated user converse with the agent and reveal information dynamically.
  • Execute tool calls against deterministic database APIs.
  • At episode end, compare database state with annotated goal state.
  • Repeat trials to compute pass^k consistency.

Claims

C1 · paper-supported

tau-bench evaluates dynamic conversations between an LM-simulated user and an agent with domain-specific API tools and policy guidelines.

Scope: Abstract and Introduction.

  • E-benchmark-setup: tau-bench emulates dynamic conversations between an LM-simulated user and an agent with domain-specific API tools and policy guidelines.

C2 · paper-supported

The evaluation compares the database state at the end of a conversation with an annotated goal state.

Scope: Abstract and Introduction.

  • E-db-state-eval: The benchmark compares final database state after a conversation to annotated goal state, allowing stochastic conversation paths that should end in the same outcome.

C3 · paper-supported

pass^k measures the consistency and robustness of agent behavior across k independent trials.

Scope: Abstract and Introduction.

  • E-passk: The paper proposes pass^k to evaluate reliability of agent behavior over multiple independent trials.

C4 · paper-supported

The first tau-bench domains are customer-service-style retail and airline tasks with databases, APIs, policies, and user scenarios.

Scope: Introduction.

  • E-domains: The first benchmark includes tau-retail and tau-airline domains with realistic databases, APIs, policy documents, and user scenarios.

C5 · paper-supported

The paper reports that state-of-the-art function-calling agents such as GPT-4o succeed on less than 50% of tasks overall.

Scope: Abstract.

  • E-overall-result: The paper says state-of-the-art function-calling agents like GPT-4o succeed on less than 50% of tasks.

C6 · paper-supported

The paper reports GPT-4o pass^1 around 61% on retail and 35% on airline, with pass^8 dropping to about 25% on retail.

Scope: Introduction.

  • E-passk-result: The paper reports GPT-4o function-calling pass^1 around 61% on retail and 35% on airline, with pass^8 on retail around 25%.

C7 · paper-supported

The paper identifies failures in database reasoning, policy following, compound requests, and consistency under stochastic conversation.

Scope: Introduction and analysis.

  • E-failures: The authors report that agents struggle with complex database reasoning, policy following, compound requests, stochasticity, and partial information.

Agent Infrastructure Takeaways

  • Production agent evaluation needs users, not just tools. Missing information must be gathered through conversation.
  • Policy compliance should be tested against state changes, not only against plausible natural-language explanations.
  • Final database state is the right verifier for many business agents because tool calls are side effects.
  • pass^k is a better reliability signal than pass@1 when small stochastic differences can break a workflow.
  • Agents need durable state, policy retrieval, authorization, idempotency, and rollback because tau-bench tasks mutate databases.
  • Simulated users are benchmark infrastructure; their fidelity and stochasticity affect the measured agent.

Caveats

  • The users are LM-simulated, not actual humans.
  • The first benchmark covers retail and airline, not all real-world domains.
  • Database-state matching can miss conversational quality unless quality is encoded in the task.
  • pass^k can be expensive because it multiplies trials per task.
  • The reported GPT-4o numbers are historical and setup-specific.
  • Strong agents could overfit to policy documents or simulator quirks if benchmark splits are not managed carefully.