Cline Bench
https://github.com/cline/cline-bench- Category
- Developer Tools
- Rank
- No. 1630Tools index
Previous survey · No. 1637 ·
- Listed in
- #11 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Builder
- cline
- GitHub
- 36 stars
- Date
About
Real-world coding benchmarks derived from actual Cline user sessions: verified, challenging engineering problems.
What it does
Cline Bench packages software repair challenges as isolated environments with task instructions, resource limits, and automated checks. Harbor runs a coding agent against one task or the full collection, using local Docker or Daytona cloud sandboxes. Each trial records configuration, conversation logs, commands, timing, costs, test output, and a binary result.
Why it's ranked here
This is a credible evaluation harness because it tests working code inside controlled environments and preserves detailed run artifacts for diagnosis. The included checks span migrations, concurrency failures, desktop packaging, C++ editing behavior, health data loaders, and numerical boundary bugs. The verdict is positive for comparative agent testing, tempered by early-access status and coarse all-or-nothing scoring.
What's good
Tasks verify concrete engineering outcomes, including successful compilation, malformed-data handling, concurrency behavior, generated assets, and numerical edge cases. Several suites explicitly avoid prescribing one implementation. Local and cloud execution use the same Harbor workflow, while provider-prefixed model names make comparisons practical across Anthropic, OpenRouter, OpenAI, and compatible endpoints. Full agent and verifier logs help explain failures.
Tradeoffs
Setup requires Python 3.13, uv, Harbor, Docker for local runs, provider credentials, and a Daytona key for cloud runs. Cloud execution costs money, and concurrent Daytona sandbox creation can fail, requiring lower concurrency or sequential runs. Binary scoring hides partial progress because one failing check makes the entire trial fail. Some verification tests also enforce specific code structure or textual patterns, not only behavior.
How to use it well
Use it for repeatable comparisons of coding agents, model providers, prompts, or execution settings on repository-scale repair work. Validate individual tasks locally with Docker before launching cloud batches, then inspect test output and conversation logs alongside the binary result. It does not replace profiling, human code review, security analysis, or a nuanced quality rubric for partially correct solutions.
Technical notes+
README.md defines the Harbor-based CLI workflow, Python 3.13 setup, Docker and Daytona environments, provider:model-id selection, job artifacts, and binary rewards. tasks/01k6n26zm27ffa7qqbcx0prrnw-police-sync-segfault/tests/test_segfault_fix.py checks malformed coordinates, threaded execution, performance, and geometry robustness. tasks/01k6zz0nyj31znwsevx4sn6zb2-telegram-plugin-refactor/tests/test_refactoring.py mixes compilation checks with required file structure and string matching. tasks/01k8ywgx6x7swdcse588426wc5-aenet-pytorch-pbc-neighborlist/tests/test_pbc_forces.py exercises imports, the upstream pytest suite, and cutoff-boundary behavior. tasks/01k8tymr1s3ndn1rzsrzy6dnfm-healthchain-prefetch-removal/tests/test_mimic_loader.py and tasks/01k8tymr1s3ndn1rzsrzy6dnfm-healthchain-prefetch-removal/tests/test_synthea_loader.py cover validation, sampling, malformed records, grouping, and missing inputs.
Observed
- Interface
- Harbor command-line benchmark runner
- Installation surface
- Python 3.13 environment with uv-installed Harbor
- Local execution
- Docker sandboxes
- Cloud execution
- Daytona sandboxes using an API key
- Provider support
- Anthropic, OpenRouter, OpenAI, and OpenAI-compatible endpoints
- Verification
- Per-task pytest suites with binary pass or fail rewards
- Task structure
- Each task includes instructions, Harbor configuration, a broken container environment, and tests
Read from README.md, tasks/01k8tymr1s3ndn1rzsrzy6dnfm-healthchain-prefetch-removal/tests/conftest.py, tasks/01kbb2wvw29szdjwcs76265t3k-filmarchiver/tests/test_film_archiver_fixes.py, tasks/01k6n26zm27ffa7qqbcx0prrnw-police-sync-segfault/tests/test_segfault_fix.py, tasks/01k6zz0nyj31znwsevx4sn6zb2-telegram-plugin-refactor/tests/test_refactoring.py, tasks/01k8251zmv88p0hztas8htr6hw-orpc-client-migration/tests/test_orpc_migration.py, tasks/01k8mwgj1z6kr0a7q59r6ek2ar-v-edit-workspace-tests/tests/test_workspace_fixes.py, tasks/01k6kr5hbv8za80v8vnze3at8h-every-plugin-api-migration/tests/test_api_migration.py, tasks/01k8tymr1s3ndn1rzsrzy6dnfm-healthchain-prefetch-removal/tests/test_mimic_loader.py, tasks/01k8ywgx6x7swdcse588426wc5-aenet-pytorch-pbc-neighborlist/tests/test_pbc_forces.py, tasks/01k8tymr1s3ndn1rzsrzy6dnfm-healthchain-prefetch-removal/tests/test_synthea_loader.py.
What it can do
Generate coding benchmarks from user sessions
Cline user session data → Verified coding benchmark problems
Verify engineering problem validity
Raw coding problems from sessions → Validated benchmark challenges
Extract real-world coding scenarios
Historical user interactions → Practical programming test cases
Curate challenging engineering problems
Pool of potential coding challenges → Filtered difficult benchmark problems
Provide benchmark evaluation framework
Code solutions to benchmark problems → Performance assessment results
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.