Why It Matters
For agent infrastructure, ToolEmu is a practical testing layer between prompt-only evaluation and real-world side effects. It does not replace real sandboxes, but it helps find dangerous trajectories early.
Problem
- Tool-using agents can leak data, transfer money, delete files, or trigger physical-world effects.
- Testing these risks manually requires tool implementations, sandbox setup, scenario design, and human trajectory inspection.
- Many high-stakes or future tools are difficult to instantiate safely in real environments.
- Long-tail ambiguous instructions are especially hard to cover with hand-built test cases.
Key gap: The paper asks how to test diverse, high-stakes agent-tool scenarios at scale without implementing every tool and sandbox.
Method
Replace expensive real tool sandboxes with an LM-emulated environment, run agents against curated risky scenarios, and use LM-based evaluators plus human validation to measure realistic failures.
- Formalize tool-using LM agents as systems that act in a partially observable environment.
- Use an LM emulator to generate tool observations from tool specifications and agent inputs.
- Add an adversarial emulator to instantiate challenging sandbox states for red-teaming.
- Build LM-based safety and helpfulness evaluators for scalable trajectory scoring.
- Validate the emulator and evaluators against human annotations.
- Curate a benchmark of 36 high-stakes toolkits and 144 test cases across 9 risk types.
- Evaluate current LM agents and prompt variants for safety-helpfulness tradeoffs.
Claims
C1 · paper-supported
ToolEmu uses an LM to emulate tool executions and sandbox states from tool specifications and tool inputs.
Scope: Abstract and Sections 1-3.
- E-emulated-sandbox: ToolEmu emulates tool execution and sandbox states with an LM using only tool specifications and tool inputs, avoiding manual implementation for many scenarios.
C2 · paper-supported
The framework pairs the emulator with LM-based safety and helpfulness evaluators to score complete agent trajectories.
Scope: Sections 1 and 3.2.
- E-evaluators: The framework includes an LM-based safety evaluator for risky actions/severity and a helpfulness evaluator for task completion.
C3 · paper-supported
The initial benchmark covers 36 high-stakes toolkits, 144 test cases, and 9 risk types.
Scope: Section 3.3.
- E-benchmark: The benchmark contains 36 high-stakes toolkits, 144 test cases, and 9 risk categories including privacy breach, financial loss, data loss/corruption, safety hazards, and computer security compromise.
C4 · paper-supported
Human validation found most sampled emulations realistic enough for risk testing, and 68.8% of failures identified by ToolEmu were judged valid real-world agent failures.
Scope: Abstract, Section 1, and Table 3.
- E-human-validation: The paper reports that over 80% of sampled tool execution trajectories were judged realistic by humans and that 68.8% of failures identified by the emulator/evaluator were validated as genuine risky failures with realistic trajectories.
C5 · paper-supported
In the benchmark, GPT-4 and Claude-2 were the strongest base agents, but still failed in 39.4% and 44.3% of test cases respectively.
Scope: Table 4 and Section 5.
- E-base-agent-results: Among base agents, GPT-4 and Claude-2 had the strongest safety/helpfulness scores, while still exhibiting failures in 39.4% and 44.3% of ToolEmu test cases.
C6 · paper-supported
Adding explicit safety requirements to GPT-4 improved both safety and helpfulness, but the best prompted agent still failed in 23.9% of test cases.
Scope: Table 4 and Section 5.
- E-safety-prompt: A GPT-4 agent prompted with safety requirements improved over the base agent and was the safest evaluated agent, but still failed in 23.9% of test cases.
C7 · inference
ToolEmu suggests a useful safety workflow: use cheap LM-emulated red-teaming to discover candidate failures, then reproduce high-severity findings in real sandboxes before shipping.
Scope: Application to agent infrastructure.
- E-agent-workflow: Because ToolEmu finds candidate failures cheaply but validates only probabilistically, a production process should triage LM-emulated findings into real sandbox reproduction and policy/runtime fixes.
Agent Infrastructure Takeaways
- Agent safety evaluation needs trajectory-level testing, not only single-turn model safety scores.
- LM-emulated sandboxes are useful for early risk discovery when real tools are expensive, unsafe, or not yet implemented.
- Do not confuse emulation with containment. A production agent still needs real sandboxing, scoped capabilities, audit logs, and policy enforcement.
- Safety prompting can reduce failures, but the paper's best prompted agent still failed on nearly one quarter of ToolEmu cases.
- Helpfulness should be measured alongside safety so the agent is not rewarded for refusing everything.
- High-severity ToolEmu findings should become regression tests in real tool sandboxes once the tool exists.
Caveats
- The emulator and evaluators are LMs, so they can hallucinate tool behavior or misjudge trajectory risk.
- The benchmark focuses on ambiguous or underspecified user instructions, not all agent threat models.
- Reported model failure rates are historical and tied to the evaluated model versions, prompts, and ToolEmu benchmark.
- ToolEmu is best interpreted as scalable red-team infrastructure, not as a formal safety guarantee.
- Human validation supports the method but also reveals nontrivial false positives or non-realistic failures.