Teaching open-source LLMs to wield 16,000 real-world APIs

What it does

ToolBench is a research platform for building LLMs that can plan, call, and reason about real-world tools. It collects over 16,000 REST APIs from RapidAPI, synthesizes instruction-following examples that span single-tool and multi-tool scenarios, and releases the resulting dataset along with training scripts and fine-tuned models called ToolLLaMA. The goal is to give open-source models the kind of general tool-use capability previously associated with closed APIs like ChatGPT.

The interesting bit

The project replaces standard Chain-of-Thought or ReAct annotation with a custom depth-first search based decision tree (DFSDT), which explicitly handles backtracking and failed API calls during data generation. This turns the data-creation process into an automated search problem rather than a simple prompt-and-response exercise, letting the pipeline annotate complex tasks that simpler methods miss.

Key highlights

• 16,464 real APIs across 3,451 tools, yielding 126K annotated instances with full reasoning traces and execution results.
• DFSDT annotation captures multi-step tool chains, including dead ends and recoveries, rather than linear reasoning paths.
• ToolLLaMA-2-7b-v2 is the latest fine-tuned release, trained in a multi-task fashion on the full dataset; earlier 7B and LoRA variants are also available.
• Ships with an API retriever and ToolEval harness for benchmarking open-domain tool selection.
• Data and models are Apache 2.0, though the dataset is intended strictly for research and education.

Caveats

• The entire dataset is synthetic, generated by ChatGPT (gpt-3.5-turbo-16k) and filtered by the authors; error rates or bias profiles are not quantified in the README.
• Live tool evaluation depends on RapidAPI access or a hosted backend that requires an application form; the server IP was updated in August 2024, indicating ongoing infrastructure changes.
• The README states ToolLLaMA matches ChatGPT (turbo-16k) in tool use, but the textual description offers no specific metrics or reproducibility details for that claim.

Verdict

Good fit for academics and open-source builders who need a large, structured corpus for tool-use pretraining; less ideal for practitioners who want a batteries-included, production function-calling service without managing API keys or synthetic data drift.