Vibeleaderboard
← All Intel
Intel / article

Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

Source
arxiv.org
Author
Karolina Dro\.zd\.z, Micha Heilbron
Date

Context

Following where things are and how they change through a story does not necessarily require a . In a study comparing models with 48 people, the researchers report human-level entity tracking at 410 million parameters, with larger contemporary models exceeding human performance on their naturalistic narratives.

For the human participants, accuracy declined with narrative complexity rather than length. That separates the difficulty of maintaining the story's changing state from simply reading more text. The result concerns this particular component of comprehension; it does not establish human-level language understanding overall.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Recommended reads
Comments

Checking sign-in…

Loading comments…