Why It Matters
ToolLLM treats an agent as a system over a real API marketplace: catalog ingestion, API metadata, retriever precision, execution traces, evaluator design, and search strategy are all part of the same tool-use stack.
Problem
- General instruction tuning gave open-source LLMs broad dialogue skills but weak tool-use ability.
- Earlier datasets often used limited APIs, single-tool settings, or no real API execution responses.
- Users cannot manually choose the correct API subset when the available tool pool contains thousands of real APIs.
- Linear ReAct-style reasoning can get stuck after a wrong API call because it explores only one trajectory.
Key gap: The paper asks how to build, train, retrieve, plan, and evaluate open-source LLM tool use over a large real-world REST API marketplace with both single-tool and multi-tool tasks.
Method
Collect and filter RapidAPI tools, generate single-tool and multi-tool instructions with ChatGPT, annotate executable solution paths with DFSDT, finetune LLaMA-2 7B into ToolLLaMA, retrieve candidate APIs with a neural retriever, and evaluate trajectories with ToolEval.
- Crawl RapidAPI tools and filter unreliable or low-quality APIs.
- Sample APIs and use ChatGPT to generate diverse single-tool and multi-tool instructions.
- Use ChatGPT with DFSDT to search for successful multi-step solution paths with real API calls and responses.
- Train a neural API retriever from generated instruction/API relevance pairs.
- Finetune LLaMA-2 7B on the retained instruction/solution-path pairs to produce ToolLLaMA.
- At inference, retrieve relevant APIs, let ToolLLaMA call APIs over multiple rounds, and score the resulting trajectory with ToolEval.
Claims
C1 · paper-supported
ToolBench filters RapidAPI into 3,451 high-quality tools and 16,464 APIs across 49 categories from an initial crawl of 10,853 tools and 53,190 APIs.
Scope: Section 2.1 and abstract.
- E-api-corpus: ToolBench collects 16,464 real-world RESTful APIs spanning 49 RapidAPI categories. The paper reports starting from 10,853 tools and 53,190 APIs and retaining 3,451 high-quality tools and 16,464 APIs after filtering.
C2 · paper-supported
ToolBench includes large-scale generated instructions and executable solution paths, including nearly 200k qualified instruction/API relevance pairs and 126,486 instruction/solution-path pairs.
Scope: Section 2.2 and Section 2.3.
- E-instructions-paths: The paper reports nearly 200k qualified instruction/API relevance pairs, specifically 87,413 I1, 84,815 I2, and 25,251 I3 instances, and 126,486 retained instruction/solution-path pairs after DFSDT annotation.
C3 · paper-supported
DFSDT improves solution-path annotation over ReAct by exploring multiple branches and allowing failed branches to be abandoned.
Scope: Section 2.3 and preliminary experiments in Section 3.1.
- E-dfsdt: DFSDT pass rates are 58.0 on I1, 70.6 on I2, 62.8 on I3, and 63.8 average, compared with ReAct at 37.8, 40.6, 27.6, and 35.3 average, and ReAct@N at 49.4, 49.4, 34.6, and 44.5 average.
C4 · paper-supported
ToolEval defines pass rate and win rate for tool-use trajectories and reports high agreement with human annotators.
Scope: Section 3.1 and Appendix A.5.
- E-tooleval: ToolEval uses pass rate and win rate. The paper reports 87.1% agreement with human annotators for pass rate and 80.3% agreement for win rate.
C5 · paper-supported
The neural API retriever substantially outperforms BM25 and OpenAI Ada embeddings on ToolBench retrieval metrics.
Scope: Section 3.1, Table 2.
- E-retriever: The API retriever reports average NDCG@1 78.0 and NDCG@5 84.9, compared with BM25 at 18.5 and 17.0 and Ada at 49.6 and 45.4.
C6 · paper-supported
ToolLLaMA plus DFSDT reaches near-ChatGPT tool-use performance on ToolBench and exceeds Text-Davinci-003 and Claude-2 in the paper's averages.
Scope: Section 3.2, Table 4.
- E-main-results: Average pass/win rates: ChatGPT+DFSDT 64.8/64.3, GPT-4+DFSDT 71.1/70.4, Text-Davinci-003+DFSDT 43.1/46.3, Claude-2+DFSDT 22.6/43.5, ToolLLaMA+DFSDT 66.7/60.0, and ToolLLaMA+DFSDT-Retriever 67.3/63.1.
C7 · paper-supported
ToolLLaMA generalizes to APIBench despite training on a different API and instruction domain, but oracle retrieval still matters heavily.
Scope: Section 3.3, Table 5.
- E-apibench: On APIBench, ToolLLaMA with its own retriever reports AST accuracy 16.77 on HuggingFace, 51.16 on TorchHub, and 40.59 on TensorHub. With oracle retrieval it reports 88.80, 85.88, and 88.62, while Gorilla-RS+Oracle reports 89.27, 93.01, and 94.16.
Agent Infrastructure Takeaways
- The tool registry is data infrastructure: every endpoint needs method, parameters, request body, code snippets, example responses, and quality checks.
- Multi-tool agents need execution traces, not just API-call labels. ToolBench stores thought, action, parameters, observation, and final-answer steps.
- Search strategy matters. DFSDT is a runtime planning primitive because it supports branch abandonment and recovery from bad tool calls.
- Retrieval is part of the action space. The model's reachable tools depend on retriever quality, so retriever evaluation belongs in the agent CI loop.
- LLM-as-judge can scale evaluation, but executable checks and human audits remain necessary where side effects or correctness matter.
- API temporality is unavoidable: ToolEval and ToolBench both acknowledge that RapidAPI versions, availability, responses, and reliability can change.
Caveats
- ToolBench depends heavily on ChatGPT for instruction generation, solution-path annotation, and evaluation.
- ToolEval is not a formal verifier; it approximates human judgment using a prompted model.
- The paper's strongest ToolBench main results mostly use oracle APIs, while production settings require retrieval.
- RapidAPI endpoints can disappear, slow down, change schemas, or return low-quality responses over time.
- ToolLLaMA's APIBench generalization improves dramatically with oracle retrieval, which shows retrieval remains a major bottleneck.
- The paper focuses on tool-use capability, not authentication, authorization, rate limits, rollback, or safe side-effect execution.