Vibeleaderboard
← Benchmark guide

Model results

DeepSWE

Choose a shared evaluation protocol to compare model scores. Versions, sources and configurations stay separate.

Atlas snapshot captured 10/19/2026, 10:20:28 AM UTC. This is an import timestamp, not the evaluation date. Coverage may be incomplete.

Reviewed recommendation and limits

Use as supplementary evidence for original repository engineering tasks, with the grading audit beside the result.

DeepSWE v1.1. Grade committed patches in a separate verifier container. Preserve agent, effort, repetitions and completed-trial denominator; keep v1 and later revisions separate.. Imported groups below may use different protocols; this review does not certify each run. Samaritan Research reports false-negative grading defects in at least 23 of 113 v1.1 tasks. A subsequent repair has not been verified in this review, so affected results need caution.

Publisher results Publisher v1.1 methods First-hand grading audit
← All evaluation groups

DeepSWE v1.1

Source
DeepSWE via Epoch AI
Version
v1.1
Measurement date
Not reported
Metric
% · Higher is better
Evaluation scope
Harness mini-swe-agent · effort medium · result date not supplied

Positions are withheld because at least one result lacks explicit independent provenance or a documented evaluation scope. Missing metadata is shown as unreported; matching imported metadata is not independent verification of the protocol.

  1. GPT-6 Astra

    openai/gpt-6-astra-20260903

    72.8%

    View modelEvidence aggregated

  2. Gemini 3.8 Flash

    google/gemini-3.8-flash-20260902

    71%

    View modelEvidence aggregated

  3. Claude Opus 5

    anthropic/claude-opus-5

    68.9%

    View modelEvidence aggregated

  4. Grok 4.6

    x-ai/grok-4.6-20260810

    67.5%

    View modelEvidence aggregated

  5. Gemini 3.7 Flash

    google/gemini-3.7-flash-20260813

    65.5%

    View modelEvidence aggregated

  6. Claude Fable 5

    anthropic/claude-fable-5

    65.4%

    View modelEvidence aggregated

  7. GPT-5.6 Sol

    openai/gpt-5.6-sol-20260709

    61.1%

    View modelEvidence aggregated

  8. GPT-5.5

    openai/gpt-5.5-20260423

    54%

    View modelEvidence aggregated

  9. Claude Opus 4.8

    anthropic/claude-opus-4-8

    48.7%

    View modelEvidence aggregated

  10. Claude Sonnet 5

    anthropic/claude-sonnet-5

    39.8%

    View modelEvidence aggregated

  11. Gemini 3.5 Flash

    google/gemini-3.5-flash-20260519

    37.4%

    View modelEvidence aggregated

  12. GPT-5.6 Terra

    openai/gpt-5.6-terra-20260709

    35.1%

    View modelEvidence aggregated

  13. GPT-5.6 Luna

    openai/gpt-5.6-luna-20260709

    11.3%

    View modelEvidence aggregated
DeepSWE model results | VibeLeaderboard