Vibeleaderboard
Index / tool
Visit github.com
Category
Developer Tools
Rank
No. 1481Tools index

Previous survey · No. 1460 ·

Pricing
Open Source
Type
TOOL
Use case
Model & Agent Evaluation
GitHub
34 stars
Date

About

Self-bench is an open-source tool that creates and runs coding-agent evals automatically from a repository's pull requests. Agents author and review Harbor environments generated from the PRs, the user approves each eval, and models and harnesses can be run concurrently in sandboxes. It lets teams measure agents on their own codebase instead of relying on public benchmarks that labs may optimize for, and supports connecting OpenAI and Claude subscriptions to avoid raw token costs.

At a glance

SelfBench turns merged pull requests into coding tasks with hidden tests, then compares agent setups by pass rate and cost. It is worth trying when public benchmarks tell you little about how models handle your repository.

Good fit

Engineering teams with representative merged changes and tests that can distinguish a working patch from a plausible one.

Consider something else when

Pass if you cannot run agent-generated code in isolation or your repository history cannot be shared with the services you would use.

Start here

Start with a small public repository and a few known fixes. Review the task and hidden tests, then compare two agent setups on the same cases.

Cost and requirements

The MIT-licensed self-hosted stack documents Bun and Docker with Compose. Budget for model inference and sandbox compute.

Limits to know

Results reflect the PRs and tests you chose. Check for task leakage and test quality before using a leaderboard to choose a model.

Tags

evalsbenchmarkingcoding-agentpull-requestsharboropen-source

Tech Stack

Node.jsDockerTypeScript

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.