The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning
Source
arxiv.org
Author
Garima Arya Yadav, Nilay Yilmaz, Yezhou Yang
Date
Context
The Unwritten benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → asks a model to infer a word from the sound of pen scratches and video of a moving hand, with no visible ink. This tests whether sound and motion together reveal information that cannot simply be read from the page. The study covers three writing styles. Benchmark abstract.
The authors report human accuracy above 80% for letters in the correct order, while the evaluated multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → models reached no more than 10%. Providing both sound and video often made model performance worse. These are letter-order scores on this particular task, and the result exposes a specific weakness in combining complementary sensory clues.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.