Tencent Hunyuan's EvolveScaler trips up frontier models on evolving facts
- Source
- x.com
- Date
Tencent Hunyuan released EvolveScaler, a benchmark that tests whether a model can reconstruct accurate ground truth from event logs where individual records get retracted and corrected after the fact, the way real operational data behaves. On the hardest tier, frontier models' accuracy falls to 11.3 percent, a collapse from the scores those same models post on static, unchanging benchmarks. The benchmark also shows that training on this kind of evolving data raises out-of-distribution scores by 5.25 points, suggesting the failure is not fundamental but a gap in what models have been trained to handle. The result is a useful corrective for anyone evaluating a model on log analysis, monitoring, or any task where the underlying facts change after ingestion: a model that scores well on a fixed benchmark can still fail badly the moment the data it reasons over gets revised, which is closer to how production logs actually behave than most current evaluation sets.
🚀 EvolveScaler is here. Read a 40-day RPG log. Now answer one question: if you skip the mini-boss on Day 7, do you still beat the final boss? The answer isn't in the log. You have to replay the world. That's Information Evolution — records get retracted, corrected, backfilled. The world keeps changing after you read it. So we build it backwards: define the world as an executable state machine, then render it into natural language. Code guarantees the logic. Language delivers the mess. ➡️ 117 prototypes. 159 question operators. 5 difficulty tiers. Up to ~1,200 events per sample. ➡️ 14 frontier models, hardest tier: median avg@5 falls to 11.3. ➡️ Train on it instead: +5.25 average across 8 out-of-distribution benchmarks. Check out our paper and project page. 📚 Paper: https://t.co/wmDPkSiwre 🏠 Project Page:
Context
Most reading-comprehension tests assume a document's facts stay fixed once written. Tencent's Hunyuan research team built EvolveScaler to instead test whether a model can track facts that get revised, corrected, or backfilled as a simulated timeline unfolds, similar to reading a long game log where an early event's outcome can later be overturned by something that happens afterward. To generate these scenarios at scale, researchers first define each one as an executable state machine with rules for how records change, then have a render that state machine into natural-language events, so the underlying logic stays consistent while the surface text reads like ordinary prose.
The resulting spans 117 scenario templates and 159 question types across five difficulty tiers, with up to around 1,200 simulated events in the hardest cases. Tencent reports that across 14 frontier models, median accuracy on the hardest tier falls to about 11%, and that training a model on EvolveScaler-generated data lifted its average score by 5.25 points across eight separate benchmarks it was not trained on. These are Tencent's own reported figures from the paper and thread rather than an independent replication, and the announcement does not name which 14 models were tested.
Checking sign-in…
Loading comments…