Model results
BrokenArXiv
Choose a shared evaluation protocol to compare model scores. Versions, sources and configurations stay separate.
Atlas snapshot captured 10/19/2026, 10:20:28 AM UTC. This is an import timestamp, not the evaluation date. Coverage may be incomplete.
Evaluation groups
Open a group to see its exact protocol and scores. Different configurations remain separate.
No model rankings are available in this snapshot. Reported results remain available below.
Other reported results
These reports have no model ranking. Open a group for its evidence and the reason positions are withheld.
- Claude Fable 5.12026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Claude Code 2.1.267; configuration=Claude-Fable-5.1 (low); reasoning effort low; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 10.85%; cost per problem $6.83; model configuration https://matharena.ai/models/anthropic_fable_51_low; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint anthropic/claude-fable-5.1; publisher does not expose a dated provider revision
- Claude Fable 5.12026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Claude Code; configuration=Claude-Fable-5.1 (high); Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 10.52%; cost per problem $19.57; result date not supplied; captured 2026-09-17; publisher flags model release after competition release
- Claude Opus 5.52026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Claude Code 2.1.280; configuration=Claude-Opus-5.5 (high); reasoning effort high; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 10.96%; cost per problem $2.41; model configuration https://matharena.ai/models/anthropic_opus_55_high; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint anthropic/claude-opus-5.5; publisher does not expose a dated provider revision
- DeepSeek V4.1 Flash2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=MathArena "deepcode" harness 0.3.1; configuration=DeepSeek-V4.1-Flash (Max); reasoning effort max; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 9.43%; cost per problem $0.27; model configuration https://matharena.ai/models/deepseek_deepseek_v41_flash; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint deepseek/deepseek-v4.1-flash; publisher does not expose a dated provider revision
- Gemini 3.8 Flash2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Antigravity CLI; configuration=Gemini 3.8 Flash; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 6.74%; cost per problem $0.60; result date not supplied; captured 2026-09-17; publisher flags model release after competition release
- GPT-6 Astra2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Codex 0.153.3; configuration=GPT-6 Astra (low); reasoning effort low; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 8.05%; cost per problem $0.63; model configuration https://matharena.ai/models/openai_gpt_6_astra_low; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint openai/gpt-6-astra; publisher does not expose a dated provider revision
- GPT-6 Astra2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Codex; configuration=GPT-6 Astra (max); Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 5.82%; cost per problem $2.26; result date not supplied; captured 2026-09-17; publisher flags model release after competition release
- GPT-6 Sol2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Codex 0.153.3; configuration=GPT-6 Sol (max); reasoning effort max; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 10.75%; cost per problem $1.63; model configuration https://matharena.ai/models/openai_gpt_6_sol; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint openai/gpt-6-sol; publisher does not expose a dated provider revision
- GPT-6.1 Sol2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=Codex 0.159.0; configuration=GPT-6.1 Sol (max); reasoning effort max; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 9.00%; cost per problem $0.94; model configuration https://matharena.ai/models/openai_gpt_61_sol; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint openai/gpt-6.1-sol; publisher does not expose a dated provider revision
- Grok 4.72026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=MathArena "grok" harness 1.0.40; configuration=Grok 4.7 (xhigh); reasoning effort xhigh; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 12.50%; cost per problem $8.69; model configuration https://matharena.ai/models/xai_grok_47; result date not supplied; captured 2026-10-15; publisher flags model release after competition release; publisher model endpoint x-ai/grok-4.7; publisher does not expose a dated provider revision
- Kimi K32026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=publisher-selected native harness (exact harness version unspecified); configuration=Kimi K3 (Think); Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 12.72%; cost per problem $5.18; result date not supplied; captured 2026-09-17
- Muse Spark 1.32026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=publisher-selected native harness (exact harness version unspecified); configuration=Muse Spark 1.3; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 10.52%; cost per problem $3.29; result date not supplied; captured 2026-09-17; publisher flags model release after competition release
- Qwen3.8 Max2026-08-native-harness
BrokenArXiv August 2026 · MathArena (reviewed publisher results)
Single-model result · Positions withheld
Measurement date not reported
Scope · Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifyin…
Mean rubric points normalized to percent; 56 questions; 0–3 grading, full credit requires identifying the statement as false; not binary accuracy; August 2026 dataset; harness=publisher-selected native harness (exact harness version unspecified); configuration=Qwen3.8-Max; Python and SageMath tools; no internet; budget/time-limit instructions varied across models, see methodology https://matharena.ai/arxiv_august/; 95% CI 12.04%; cost per problem $3.64; result date not supplied; captured 2026-09-17; publisher flags model release after competition release