Vibeleaderboard
Index / tool
Visit deepswe.datacurve.ai
Category
Developer Tools
Rank
No. 1075Tools index

Previous survey · No. 1041 ·

Pricing
Free
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

A coding benchmark from Datacurve built to resist contamination: its 113 tasks are written from scratch rather than adapted from existing commits or pull requests, span 91 repositories and five languages, and are graded by hand-written verifiers that test behaviour rather than implementation. Every model runs on the same mini-swe-agent harness, and reference solutions are roughly 5.5x more code than comparable benchmarks. The leaderboard covers 24 frontier models, with the task data and harness code published.

What it does

DeepSWE measures how far an AI coding agent gets on long feature requests inside real open-source projects such as Helm, FastAPI, httpx and go-git. Each task pins a project at a fixed commit, hands the agent a dense written spec, and gives it hours in a sealed container with no network. When the agent finishes, its commits are pulled out as a patch and checked in a fresh container by tests the agent never saw, which check what the code does rather than how it is named or structured.

Why it's ranked here

The task specs are unusually demanding and specific. The Helm task asks for chart-scoped array merge strategies, global value handling, upgrade semantics and lint warnings in one change; the go-git task asks for a full three-way merge with conflict staging. Grading in a separate clean container closes the trick of tampering with tests in place. All task data and the harness are public under Apache 2.0, and every upstream project is listed with its permissive license. That makes the results inspectable and reproducible, which most vendor-published coding scores are not.

What's good

Isolation is concrete: both the agent and the grader run with networking off, each container gets two CPUs and eight gigabytes of memory, and the agent has a three hour ceiling. Each run produces a binary pass plus pass fractions and a machine-readable failure report, so a near miss is visible rather than flattened to zero. The provenance table spells out that the license covers only Datacurve's own specs, verifiers and curation, not the upstream code. A seeded random subset option lets you run ten tasks cheaply before committing to all 113.

Tradeoffs

Published scores come from one minimal agent scaffold, so they rank models under that harness, not the products people actually use; the runner can drive other agent CLIs, but those runs are yours to do. The published task format ships a reference solution beside each prompt, and the prompts themselves are public, so the contamination resistance depends on models not training on this repository from here on. Running it requires the companion runner, prebuilt container images from a public registry, and API keys, and a full pass at three hours per task is a real compute bill.

How to use it well

Use it to compare models or agent setups on multi-file feature work in mature codebases, the kind of job a senior engineer scopes in a ticket. Start with the seeded ten-task subset to check your harness, then run the full set under the same scaffold behind the published scores before comparing against them. Read the per-test failure reports, not just the pass rate. It does not cover bug triage, code review, front-end visual work or interactive pairing, so pair it with other suites for those.

Technical notes+

Per README.md, tasks follow the Harbor layout (task.toml, instruction.md, environment, tests, solution) and are run with Pier newer than 0.3.0. In tasks/fastapi-implicit-head-options/task.toml and tasks/helm-array-merge-strategies/task.toml, both [agent] and [verifier] set network_mode = "no-network", the verifier uses environment_mode = "separate" with timeout_sec 1800, the agent gets timeout_sec 10800, and a [[verifier.collect]] hook runs git diff --binary against the pinned base_commit_hash into an artifact patch. docker_image points at a public ECR repository. tasks/manifest.schema.json fixes source_dataset to "swe-bench-ultra" and requires a category and language per task; both sampled tasks are category feature_request. tasks/dataset.toml names the dataset datacurve/deep-swe-1-1 with sha256 digests per task. PROVENANCE.md maps each task id to its upstream repo and license. Each instruction.md ends with a request to work on a new branch and commit.

Observed

License
Apache-2.0 for Datacurve's own specs, harness, verifiers and curation; upstream code keeps its own permissive license (PROVENANCE.md)
Task corpus
113 tasks across TypeScript, Go, Python, JavaScript and Rust (README.md)
Task format
Harbor task format: metadata, prompt, Dockerfile environment, held-out tests, reference solution
Runner
Pier CLI installed with uv; drives mini-swe-agent, claude-code, codex, gemini-cli and opencode, locally or on Modal
Sandbox
Agent and verifier both set to no-network; 2 CPUs and 8192 MB memory per container (task.toml)
Verification
Agent commits extracted as a patch and verified in a separate pristine container
Verifier output
Binary reward plus pass fractions, CTRF test report, raw logs
Published-score harness
All published scores produced with Pier running mini-swe-agent on Modal

Read from README.md, LICENSE, PROVENANCE.md, tasks/README.md, tasks/dataset.toml, tasks/manifest.schema.json, tasks/fastapi-implicit-head-options/instruction.md, tasks/fastapi-implicit-head-options/task.toml, tasks/helm-array-merge-strategies/instruction.md, tasks/helm-array-merge-strategies/task.toml, tasks/go-git-worktree-merge-conflicts/instruction.md.

Intel on DeepSWE

More in Intel

Tags

benchmarkcoding-agentsevaluationleaderboardswecontamination

Media

DeepSWE

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.