Meta previews WildArtifactBench, scoring agents by preference Elo

WildArtifactBench, which Meta describes as an internal evaluation framework, is built for agent work that has no single correct answer. Instead of grading each output against a fixed answer key or a strict rubric, it has human and AI-agent judges compare deliverables and say which they prefer. Those preferences are summarized as win rates and Elo scores. Elo is the rating system from chess: each head-to-head comparison raises the winner's rating and lowers the loser's, so after many comparisons the ratings put entries in order. According to Meta's post, the point of this design is coverage: dropping the answer key broadens the range of practical multimodal workflows the benchmark can include.

So far the release is a preview. Meta published ten tasks rather than the full benchmark. The preview page lists them by name and category: bird-sound audio analysis, cardiac ultrasound, coronary angiography video analysis, a DQN driving task filed under machine learning engineering, materials metrology from images, 3D modeling of a microchip, video editing, and digital design tasks such as a kitchen layout and a cat mesh. The page's task viewer lists instruction, data and rubric files for each task, but neither the post nor the preview page, as read here, gives any scores, says which agents were compared, or explains how human and agent judgments are combined. Readers can see the kinds of tasks involved but not how any system performs on them. The post's "also" points to a companion announcement, likely Meta's Muse Spark 1.2 evaluation thread, but it doesn't say whether WildArtifactBench was used on that model.