Intel / article

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Source: arxiv.org
Author: Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu
Date: Published Aug 31, 2026

Context

Success at individual phone actions does not establish that an AI assistant can finish a complicated request. The GMA benchmark tests 300 tasks across four difficulty tiers, from single actions to realistic requests that span four to seven apps. It reports two measures: success rate, which counts a task only when every subtask is met, and an average score, the share of subtasks completed.

Across eight evaluated models, none fully completed more than 20% of the hardest, realistic-tier tasks. The gap between the two measures is visible in the table: Claude Opus 4.7 scored about 45% on average subtask completion in that tier but fully succeeded on 15% of tasks. The authors conclude that current agents remain far from reliably handling realistic user requests. These are results in seven purpose-built apps, not observations of consumer deployment.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Read the source arxiv.org