SkillsBench
www.skillsbench.ai- Category
- AI Agents
- Rank
- No. 2806Tools index
Previous survey · No. 2709 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web · API
- Builder
- benchflow-ai
- GitHub
- 1.9k stars
- Latest release
- v1.1
- Date
About
The first benchmark framework for evaluating AI agent skills across 84 diverse tasks and 7 models. It measures how well AI agents perform when equipped with domain-specific skills versus without them, providing structured evaluation across multiple abstraction layers.
What it does
SkillsBench is a test suite that asks one question: does handing an AI agent a folder of reusable instructions and scripts actually make it better at real work? Each task ships a prompt, a pinned Docker environment, a bundle of skills, a human-written reference solution and a checker that scores the result as pass or fail. You run the same task twice through the BenchFlow command line, once with the skills injected and once without, and compare. Tasks run in a Modal cloud sandbox by default or in local Docker.
Why it's ranked here
The design isolates the variable most agent benchmarks blur. Skills are injected at run time rather than baked into the image, prompts are forbidden from naming them, and the reference solution must score a full pass before any agent runs. Categories come from a written codebook with eight controlled domains, separate labels for operation, file type and interface, and a linter that enforces them on every pull request. Apache 2.0 licensed, with the dataset also published on Hugging Face. For anyone deciding whether skill folders are worth writing, this is a measured answer rather than a vibe.
What's good
The contributor rules are specific and enforceable. Checkers must test the produced output, not which tools the agent used, and must not need paid API keys. Reference solutions must compute the answer instead of hardcoding it. Skills must be reusable beyond one task. Continuous integration rejects any task missing its solve or test entry scripts, not just an empty folder. There is also an agent-to-agent server that can drive seven harnesses, including Claude Code, Codex, OpenHands and Gemini CLI, and it redacts API-key-shaped strings from what it returns.
Tradeoffs
Numbers disagree across sources: the catalogue blurb says 84 tasks, the contributing guide cites an 87-task roster, and the guide calls the release 1.2 while pinning a CLI line it labels 1.1. The agent-to-agent server calls itself a skeleton. Its Codex and Gemini launch templates switch off approval prompts and sandboxing, so run it only somewhere disposable. The checked-in experiment config points at one developer's home directory and must be rewritten before reuse. Cloud runs need Modal tokens, agent runs need model API keys, and the linter covers only changed Python files, leaving existing lint debt in place.
How to use it well
Best fit: teams writing agent skills who want evidence that a skill helps before shipping it, and researchers comparing harnesses. Start with a default task, which needs no external credentials, run the reference solution locally in Docker, then run your agent with and without skills and read the trajectories, not just the pass rate. Contributing one strong task can earn co-authorship consideration on the paper. It does not measure general coding ability or cost, and credentialed GPU tasks sit in a separate folder you must opt into.
Technical notes+
Python 3.12+; pyproject.toml pins benchflow[sandbox-daytona] >=0.6.3,<0.7 and a2a-sdk 0.3.20. CONTRIBUTING.md defines the task package: a prompt file with YAML frontmatter (metadata, timeouts, network_mode), an environment folder holding the Dockerfile and skills, an oracle solve script and a verifier test script that writes a 1 or 0 reward. .github/scripts/validate_tasks.py rejects the older split layout and requires both entry scripts; .github/workflows/ci.yml runs it plus skill frontmatter linting and ruff on changed files. skillsbench_agentbeats/server.py serves an A2A green agent on 127.0.0.1:9009; skillsbench_agentbeats/agent_under_test.py defaults to openhands with a 900s timeout, truncates final text to 4000 chars and caps materialized files at 1,000,000 bytes. experiments/configs/main-with-skills.yaml sets 5 attempts and 256 concurrent trials.
Observed
- License
- Apache 2.0
- Language
- Python, requires 3.12 or newer
- Install surface
- BenchFlow CLI installed as a uv tool; repository tooling from a committed uv lockfile
- Execution sandboxes
- Modal cloud sandbox by default; local Docker supported
- Interfaces
- bench command line for task checks and eval runs; A2A agent server for AgentBeats
- Supported agent harnesses (A2A participant)
- OpenHands, opencode, Claude Code, Codex, Gemini CLI, Terminus, pi
- Task package
- Prompt with YAML frontmatter, Dockerfile environment, bundled skills, oracle solution script, verifier test script
- Dataset distribution
- Hugging Face dataset
- CI checks
- Task structure validation, skill frontmatter lint, ruff on changed Python files
Read from README.md, pyproject.toml, LICENSE, CONTRIBUTING.md, docs/instruction-guidelines.md, docs/unit-test-guidelines.md, docs/harnesses/README.md, skillsbench_agentbeats/server.py, skillsbench_agentbeats/executor.py, skillsbench_agentbeats/agent_under_test.py, taxonomy.md, .github/scripts/validate_tasks.py, .github/workflows/ci.yml, experiments/configs/main-with-skills.yaml.
What it can do
Evaluate AI agent performance across multiple tasks
AI agent and selected task set → Performance metrics and scores
Compare AI agent performance with and without domain-specific skills
AI agent, domain skills, and benchmark tasks → Comparative performance analysis
Benchmark AI agents across 84 diverse tasks
AI agent model → Comprehensive task performance report
Assess AI agent capabilities across 7 different models
Multiple AI agent models → Cross-model performance comparison
Measure skill effectiveness across multiple abstraction layers
AI agent with skills and abstraction layer specifications → Layer-specific skill effectiveness metrics
Generate structured evaluation reports
Benchmark test results → Formatted evaluation report
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.