ToolBench
https://github.com/openbmb/toolbench- Category
- AI Tools
- Rank
- No. 1022Tools index
- Listed in
- #20 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Builder
- openbmb
- GitHub
- 5.7k stars
- Date
About
OpenBMB's open platform for training, serving, and evaluating LLMs for tool learning — over 16K real-world APIs in the training set.
What it does
ToolBench is a research stack for teaching language models to select and call external tools. It supplies generated instructions, recorded tool interactions, retrieval data, model training, inference, and evaluation workflows for both single-tool and multi-tool tasks.
Why it's ranked here
The project combines data generation, retriever training, model fine-tuning, inference, and evaluation in one reproducible research workflow. Its strongest case is experimental coverage. Its weakest is operational reliability, since real API access introduces subscriptions, rate limits, authorization failures, timeouts, and unavailable services.
What's good
The dataset records reasoning, tool calls, execution results, and final answers instead of only prompt-response pairs. Separate scenarios test one tool, multiple tools within a category, and tools across collections. The training pipeline masks prior conversation content so loss targets the final assistant response.
Tradeoffs
This is explicitly for research and education, not a production tool platform. Setup requires separately downloaded data and a large Python dependency set. The documented full-model example uses two 80 GB A100 GPUs. Live API experiments can fail because of authentication, subscriptions, quotas, gateways, or provider downtime.
How to use it well
Use it to study tool selection, multi-step API planning, retriever training, supervised fine-tuning, and comparative evaluation. Start from the prepared datasets and released model when reproducing experiments. Choose a separate serving and reliability layer when you need stable production execution rather than research infrastructure.
Technical notes+
preprocess/preprocess_retriever_data.py splits query data and emits JSON, TSV, and text artifacts for retrieval training. preprocess/preprocess_toolllama_data.py converts recorded assistant messages into Thought, Action, and Action Input training conversations. toolbench/train/train.py tokenizes formatted conversations, masks non-target tokens, and offers both eager preprocessing and LazySupervisedDataset, which preprocesses examples on first access and stores results in an in-memory dictionary. toolbench/train/train_mem.py applies a FlashAttention monkey patch before training, while toolbench/train/train_lora.py adds PEFT LoRA training. toolbench/inference/utils.py implements token streaming and sampling controls. toolbench/inference/server.py constructs imports and argument expressions dynamically, then executes them with exec and eval.
Observed
- License
- Apache License 2.0 for the released dataset
- Primary language
- Python
- Runtime requirement
- Python 3.9 or newer
- Installation surface
- Source clone followed by pip installation from requirements.txt; ToolEval has a separate requirements file
- Interfaces
- Command-line preprocessing, retriever training, model training, and inference scripts, plus a documented web demo
- Model stack
- PyTorch, Transformers, Sentence Transformers, DeepSpeed, PEFT, FastAPI, and Gradio
- Released artifacts
- Instruction and answer data, retrieval data, test splits, ToolLLaMA models, and a trained tool retriever
Read from README.md, requirements.txt, toolbench/utils.py, toolbench/tool_conversation.py, preprocess/preprocess_retriever_data.py, preprocess/preprocess_toolllama_data.py, toolbench/train/train.py, toolbench/model/__init__.py, toolbench/tooleval/utils.py, toolbench/inference/utils.py, toolbench/retrieval/train.py, toolbench/train/train_mem.py, toolbench/inference/server.py, toolbench/model/make_delta.py, toolbench/train/train_lora.py.
What it can do
Train LLMs for tool learning
Large language models and real-world API datasets → Tool-learning capable LLM models
Serve trained LLM models
Trained tool-learning models → API endpoints for model inference
Evaluate LLM tool learning performance
LLM models and evaluation datasets → Performance metrics and benchmarks
Provide access to real-world API training data
User request for API datasets → Curated dataset of 16K+ real-world APIs
Generate API usage examples
API specifications and use cases → Code examples and implementation patterns
Benchmark tool learning models
Multiple LLM models and standardized test cases → Comparative performance reports
Intel on ToolBench
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.