ARC Prize's public leaderboard announcement created a way to test internet-connected models outside its more restrictive competition. That makes experimentation easier, but changes how the results should be interpreted: public puzzles and solutions may have appeared in training data or influenced how a system was built.
The organizers therefore reported scores on a separate verification set as well. They also found that the evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → sets did not appear equally difficult, so a lower verification score could not automatically be blamed on memorization. The useful comparison needs both the test conditions and the dataset, rather than treating every score carrying the ARC-AGI name as interchangeable.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
