Vibeleaderboard
← All Intel
Intel / post

Vals AI Launches CUA-Bench for Real-Time Computer Use

Source
x.com
Date
Vals AI@ValsAI

What tasks are left that humans find easy but today's models still find hard? Two such tasks are computer use and games. We’re launching CUA-Bench, a benchmark testing how well AI can use a keyboard and mouse across 6 games (3 kept private) in real time. To saturate it, models will need to output real-time actions and learn continuously from video, not just text.

Context

Vals AI has launched CUA-Bench, a that tests how well an AI model can play six games by working a keyboard and mouse in real time. Three of the six games are kept private. Vals AI chose computer use and games as examples of tasks that humans find easy and today's models still find hard.

The real-time requirement is the hard part the benchmark is built around. A game does not pause while a model decides what to press, so a model has to produce its actions quickly enough to keep up with what is happening on screen. Vals AI says that to saturate CUA-Bench, meaning to reach scores near the top of the scale, models will need to output actions in real time and learn continuously from video, not just from text. That is a statement about what the benchmark is designed to demand, not a result.

The launch post's text contains no scores, so it says nothing yet about how current models perform. It also does not name the games, explain how a run is scored, or say how the three private games are used. For now CUA-Bench is a new yardstick for a capability that Vals AI says today's models still find hard, and the evidence about any particular model will come when results are published.

Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
More from Vals AI
Recommended reads
Comments

Checking sign-in…

Loading comments…