agent-readings

Tool Use and Interoperability

Gorilla

Gorilla turns API use into a retrieval-aware function-calling problem: pick the right API from a large changing catalog, generate a valid call, and avoid hallucinated tools.

6Claims
7Evidence

Why It Matters

Gorilla makes API documentation and catalog freshness part of the agent stack. The model is not only choosing an action; it is choosing from a mutable registry whose entries, arguments, versions, and constraints change over time.

Problem

  • Prompted LLMs could call small, well-described tool sets, but the prompt context cannot hold web-scale API catalogs.
  • State-of-the-art models such as GPT-4 still made wrong API calls or hallucinated nonexistent API usage.
  • API calls are hard to evaluate with ordinary unit tests because many APIs can be functionally similar and many arguments are optional.

Key gap: The paper asks how to train and evaluate an LLM that maps natural-language goals to correct API calls across a large, overlapping, frequently updated API corpus.

Method

Collect API documentation from model hubs, generate instruction/API pairs with self-instruct, finetune a LLaMA-7B model into Gorilla with and without retrieved documentation, and evaluate generated calls using AST subtree matching plus hallucination/error accounting.

  • Curate API documentation from TorchHub, TensorHub, and HuggingFace.
  • Generate synthetic user instructions and reference API answers with self-instruct.
  • Convert each example into a one-turn user/agent instruction-tuning conversation.
  • Optionally append retrieved API documentation to the user prompt for retriever-aware training.
  • At inference, run Gorilla zero-shot or append top retrieved API documentation.
  • Evaluate the generated API call by AST subtree matching against the API database.

Claims

C1 · paper-supported

APIBench covers 1,645 ML APIs across TorchHub, TensorHub, and HuggingFace, with 10 generated instructions per API for 16,450 instruction/API pairs.

Scope: Section 3.1 and Appendix A.1 in the accepted paper.

  • E-dataset: The accepted paper describes 1,645 API calls: 94 from TorchHub, 626 from TensorHub, and 925 from HuggingFace, with 10 generated instructions per API, yielding 16,450 instruction/API pairs.

C2 · paper-supported

Gorilla is a LLaMA-7B model instruction-finetuned for API calls, with retriever-aware training that teaches it to use or ignore retrieved API documentation.

Scope: Section 3.2 and Figure 3.

  • E-method-rat: Gorilla is a retriever-aware finetuned LLaMA-7B model. During RAT, the prompt includes retrieved API documentation, and the model is trained to use relevant documentation or fall back to learned API knowledge when retrieval is irrelevant.

C3 · paper-supported

The paper evaluates API-call correctness with AST subtree matching and defines hallucination as a generated API call that matches no API in the database.

Scope: Section 3.3 and Figure 4.

  • E-ast: Generated API calls are parsed into ASTs and matched against reference API subtrees. A sampled manual check reports 78% AST accuracy, 78% human-evaluated API accuracy, and 72% executable supporting code.

C4 · paper-supported

In zero-shot API generation, Gorilla reports higher overall accuracy and lower hallucination than GPT-4 across TorchHub, HuggingFace, and TensorHub in Table 1.

Scope: Section 4.1 and Table 1.

  • E-main-results: In zero-shot mode, Gorilla scores 59.13 TorchHub, 71.68 HuggingFace, and 83.79 TensorHub overall accuracy with hallucination rates 6.98, 10.95, and 5.40. GPT-4 zero-shot scores 38.70, 19.80, and 18.20 accuracy with hallucination rates 36.55, 37.16, and 78.65.

C5 · paper-supported

Retriever-aware training can adapt to test-time documentation changes and improves with strong retrievers, but weak retrieval at inference can sharply degrade accuracy.

Scope: Section 4.1, Section 4.2, Figure 6, and Table 2.

  • E-retrieval: With oracle retrieval, RAT-trained Gorilla reaches 67.20 TorchHub, 91.26 HuggingFace, and 94.16 TensorHub overall accuracy. The paper also reports that using non-oracle retrieval at evaluation can produce large degradation, including 29.20% for GPT-Index and 52.27% for BM25 in one comparison.
  • E-doc-change: The accepted paper shows Gorilla responding to test-time documentation changes such as upgrading FCN from ResNet-50 to ResNet-101 and changing the model registry from pytorch/vision to NVIDIA/DeepLearningExamples:torchhub.

C6 · paper-supported

Gorilla focuses on selecting and generating single API calls for ML APIs; API execution, multi-step tool workflows, and general web API side effects are outside the main paper scope.

Scope: Methodology, inference note, Appendix dataset/task definition, and limitations/social impact section.

  • E-scope-limits: The paper says API execution exists but is not the focus, defines the API-call task as generating a single line of code or leveraging an API reference, and notes that the dataset is ML API focused.

Agent Infrastructure Takeaways

  • API catalogs are first-class infrastructure. Agent quality depends on normalized docs, versioning, retriever freshness, and schema fields for arguments and constraints.
  • Function-calling evaluation needs structural checks. AST subtree matching separates invented APIs from wrong-but-existing APIs more cleanly than text similarity.
  • Retrieval is not automatically helpful. A runtime must measure retriever precision and teach models when to ignore irrelevant retrieved documentation.
  • Hallucination should be typed. Gorilla distinguishes nonexistent tools from incorrect API choices, which maps naturally to different remediation paths.
  • Constraint handling belongs in the API schema: performance, parameter count, latency, cost, and requirements need to be queryable if agents must choose among tools.
  • This is a tool-selection layer, not an execution runtime. Side effects, permissions, retries, rollback, and multi-step plans still need separate infrastructure.

Caveats

  • The core benchmark is centered on ML APIs, not arbitrary REST, SaaS, or operating-system APIs.
  • The main task generates one API call or small code snippet rather than a full multi-step workflow.
  • Oracle retrieval is an upper bound; BM25 and GPT-Index retrieval can degrade accuracy when irrelevant docs are appended.
  • AST matching checks structural API-call equivalence and does not prove complete program execution for every output.
  • For HuggingFace, non-Gorilla baseline evaluation is relaxed because the API set is not exhaustive.
  • The work does not solve side-effect safety, authentication, authorization, or sandboxing for real API execution.