Vibeleaderboard
← All Intel
Intel / article

AppEval: A Unified Benchmark for LLM-Based Mobile Application Repair in ArkTS, Swift, and Kotlin

Source
arxiv.org
Author
Bang Xie, Hao Liu, Zhenyu Shi, Yonghao Zhang, Senjian Zhang, Zhiyuan Peng, Xin Yin, Chenhao Ying, Yuan Luo, Wei Chen, Haiming Jin, Shaocong Long, Xu Liu, Zhe Peng
Date

Context

AppEval separates a broken mobile app from a broken testing environment. A task qualifies only when the same installed-app target reaches a failing behavioral assertion in the defective version and passes after the reference fix. Missing development tools, unavailable devices and other infrastructure failures remain separate outcomes. This makes the benchmark's comparison depend on exercising the app's behavior.

The authors report first-attempt success rates ranging from 22% to 90.5% across five agents on 200 Android tasks from 24 repositories. Although the framework also covers iOS and HarmonyOS, those quantitative results apply specifically to Android.

Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Recommended reads
Comments

Checking sign-in…

Loading comments…