Cua-Bench-S1 Benchmarks Single-Decision Computer-Use Models

Cua-Bench-S1 measures a narrow kind of model: one that makes a single computer-use decision by picking from options the application has already listed, instead of an agent that plans and acts on its own. Cua introduced it alongside two checkpoints it calls the first generation of Cua-S1 models, and says it also covers decision models such as Jev, a model from TypeSafe that Cua has also paired with its desktop driver. According to the model cards, the core tasks are form filling, login, consent checkboxes, multi-step submission, pagination and search filters, built with Cua's own synthetic generator plus converters for two existing datasets, GUI-360 and AndroidControl, a Google Research collection of over 15,000 human demonstrations across 833 Android apps designed to test whether agents generalize beyond the apps and tasks they were trained on. Chess and game control serve as out-of-domain checks, and there are also safety-gate tasks and an external general-decision benchmark.

The two checkpoints sit at opposite ends of size. Cua-S1-Nano-0.1 is a classifier of about 855,000 parameters, trained from scratch, that scores every option in one parallel pass, which Cua puts at low milliseconds per task on a GPU. Cua-S1-4B-0.1 is a pair of LoRA adapters on a frozen copy of Qwen's Qwen3.5-4B that reads the screen as an accessibility tree or a screenshot and picks one lettered option.

The first results show how far both are from general use. Each scores 1.000 on the multimodal split drawn from the same distribution it was trained on. On held-out text tasks from other datasets, task accuracy falls to 0.000 to 0.286 for Nano and 0.167 to 0.571 for the 4B model, depending on the task family, and both score zero on chess and game control, results Cua says do not survive chance correction. Cua's own reading of Nano is that it mostly reflects its training distribution and is not a general interface decision model.

Cua now points new work to Cua-S1-4B-0.2, which the 0.1 card says scores higher on the held-out and live-environment evaluations. The same card records local tests on September 25 in which 4B-0.1 failed two of three synthetic requests, so the first generation is best read as a research baseline for comparison rather than something to run on real accounts.