Model results
DeepSWE
Choose a shared evaluation protocol to compare model scores. Versions, sources and configurations stay separate.
Atlas snapshot captured 10/19/2026, 10:20:28 AM UTC. This is an import timestamp, not the evaluation date. Coverage may be incomplete.
Reviewed recommendation and limits
Use as supplementary evidence for original repository engineering tasks, with the grading audit beside the result.
DeepSWE v1.1. Grade committed patches in a separate verifier container. Preserve agent, effort, repetitions and completed-trial denominator; keep v1 and later revisions separate.. Imported groups below may use different protocols; this review does not certify each run. Samaritan Research reports false-negative grading defects in at least 23 of 113 v1.1 tasks. A subsequent repair has not been verified in this review, so affected results need caution.
Evaluation groups
Open a group to see its exact protocol and scores. Different configurations remain separate.
No model rankings are available in this snapshot. Reported results remain available below.
Other reported results
These reports have no model ranking. Open a group for its evidence and the reason positions are withheld.
- DeepSeek V4.1 Flashv1.1
DeepSWE v1.1 · DeepSeek model card (vendor-reported)
Single-model result · Positions withheld
Measurement date 2026-09-10
Scope · Vendor-reported; reasoning effort 100; Resolved; temperature 1.0, top_p 0.95; mini-SWE; 1M context
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
17 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · effort high · result date not supplied
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
11 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · effort low · result date not supplied
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
12 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · effort max · result date not supplied
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
13 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · effort medium · result date not supplied
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
13 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · effort xhigh · result date not supplied
- DeepSWE v1.1v1.1
DeepSWE via Epoch AI
3 models with scores · Positions withheld
Measurement date not reported
Scope · Harness mini-swe-agent · result date not supplied
- Muse Spark OpenVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-10-15
Scope · Registered model evaluation · Muse Spark Open model card
- MiMo-V2.6-FlashVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-09-22
Scope · Registered model evaluation · MiMo-V2.6-Flash-RL model card
- MiMo-V2.6-ProVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-09-22
Scope · Registered model evaluation · MiMo-V2.6-Pro-RL model card
- Step 5 PreviewVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-09-20
Scope · Registered model evaluation · Step 5 Preview model card
- Nex-N2.5-ProVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-09-11
Scope · Registered model evaluation · Nex-N2.5 model card
- DeepSeek V4.1 FlashVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-09-10
Scope · Registered model evaluation · DeepSeek-V4.1-Flash model card
- DeepSeek V4 Flash Vision ExpVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-31
Scope · Registered model evaluation · Model Card
- deep-swe · deep sweVersion not reported
Hugging Face eval result
2 models with scores · Positions withheld
Measurement date 2026-08-28
Scope · Registered model evaluation · Model Card
- GLM 5.3 FlashVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-26
Scope · Registered model evaluation · GLM-5.3-Flash model card
- Qwen3.8 FlashVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-26
Scope · Registered model evaluation · Qwen3.8-Flash-Next model card
- Qwen3.8 27BVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-14
Scope · Registered model evaluation · Qwen3.8-27B model card
- Qwen3.8 2.4T A95BVersion not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-12
Scope · Registered model evaluation · Model Card
- DeepSeek V4 Flash 0731Version not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-08-03
Scope · Registered model evaluation · Model Card
- Kimi K3Version not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-07-27
Scope · Registered model evaluation · Model Card
- Laguna S 2.1Version not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-07-21
Scope · Registered model evaluation · Model Card
- Hy3Version not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-07-06
Scope · Registered model evaluation · Model Card
- GLM 5.2Version not reported
deep-swe · deep swe · Hugging Face eval result
Single-model result · Positions withheld
Measurement date 2026-06-22
Scope · Registered model evaluation · Model Card