Self-bench
github.com/mupt-ai/self-bench- Category
- Developer Tools
- Rank
- No. 1481Tools index
Previous survey · No. 1460 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 34 stars
- Latest release
- prod-20261117-67ac39d9
- Date
About
Self-bench is an open-source tool that creates and runs coding-agent evals automatically from a repository's pull requests. Agents author and review Harbor environments generated from the PRs, the user approves each eval, and models and harnesses can be run concurrently in sandboxes. It lets teams measure agents on their own codebase instead of relying on public benchmarks that labs may optimize for, and supports connecting OpenAI and Claude subscriptions to avoid raw token costs.
At a glance
SelfBench turns merged pull requests into coding tasks with hidden tests, then compares agent setups by pass rate and cost. It is worth trying when public benchmarks tell you little about how models handle your repository.
Good fit
Engineering teams with representative merged changes and tests that can distinguish a working patch from a plausible one.
Consider something else when
Pass if you cannot run agent-generated code in isolation or your repository history cannot be shared with the services you would use.
Start here
Start with a small public repository and a few known fixes. Review the task and hidden tests, then compare two agent setups on the same cases.
Cost and requirements
The MIT-licensed self-hosted stack documents Bun and Docker with Compose. Budget for model inference and sandbox compute.
Limits to know
Results reflect the PRs and tests you chose. Check for task leakage and test quality before using a leaderboard to choose a model.
Sources checked 2026-11-05. Source-based assessment.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.