FrontierMath
epoch.ai- Category
- Developer Tools
- Rank
- No. 2182Tools index
Previous survey · No. 2054 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Date
About
Epoch AI's held-out, expert-written mathematics problems, ranging from advanced undergraduate through research level across tiers. Compare the tier, version, and sampling budget before comparing model scores.
What it does
Epoch AI presents FrontierMath as its program for testing AI on advanced mathematics, in three parts: a tiered benchmark of unpublished problems written by mathematicians, a collection of open research problems whose answers a computer can check, and Erdős problems stated in Lean, where the model must produce a complete formal proof or disproof.
Stated on the product site
- Components
- Tiers 1-4 benchmark, Open Problems, and FrontierMath Erdős
- Difficulty range
- Tiers 1-3 run from undergraduate to advanced graduate level; Tier 4 is research level
- Problem set
- Several hundred unpublished problems in the tiered benchmark
- Grading method (Open Problems)
- Solutions are computationally verifiable, so no human proof review is needed
- Answer format (Erdős)
- Problems are stated in Lean and require complete Lean proofs or disproofs
Not stated on the site
- How many problems each tier holds, and how many are in the Erdős and Open Problems sets.
- Whether any of the problems or a sample set can be accessed by outside researchers.
- Which models have been evaluated, and under what sampling budget or attempt limits.
- How often the problem sets are refreshed or versioned.
Written from the product site at epoch.ai.
Tags
benchmarkevaluation
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.