Cua Releases Cua-Bench-S1 and Two Open Decision-Model Checkpoints
- Source
- Cua
- Date
1/ Introducing Cua-Bench-S1, a benchmark for decision models built for computer use, including Jev. We're releasing the first generation of Cua-S1 models, with two checkpoints: Cua-S1-Nano-0.1 and Cua-S1-4B-0.1 Cua-S1-Nano-0.1: https://t.co/ep6R2H4ahL Cua-S1-4B-0.1: https://t.co/p50GSsrXiP Repo:
2/ Here is what that looks like in practice. The clip shows a model making one chess move and working through a form task, each ending on a PASS screen.
3/ In every Cua-Bench-S1 task, the decision model reads one state and picks one option from a fixed list of element-and-action candidates. It decides once with no planning loop nor retries.
4/ Four task families share that format: interface actions, safety decisions about when not to act, game control, and held-out probes. The probes are chess, Doom, and JevBench, an external benchmark we use to see how a model handles tasks it was not trained for.
- Every Cua-Bench-S1 task gives a decision model one state and a fixed list of element-and-action candidates; it picks one option once, with no planning loop or retries. Cua-Bench-S1 scores offline datasets and is separate from the live-VM Cua-Bench .
- The held-out probes use chess, Doom and JevBench, an external , to test how a model handles tasks it was not trained for. The other families cover interface actions, safety decisions about when not to act, and game control.
- Cua-S1-Nano-0.1 is an option- model trained from scratch that scores every candidate in one forward pass. Cua-S1-4B-0.1 is a on Qwen 3.5 4B that reads text plus a screenshot and returns the same single choice. Both ship with model code and training recipes.
- Cua calls this gen 0 research artifacts. Gen 1 targets cleaner train/eval splits, more open-ended tasks, larger and faster models, more training methods and more task families.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Gives builders of computer-use decision models a shared benchmark and two open baselines to measure against, rather than relying on ad hoc single-task demos.
Checking sign-in…
Loading comments…