AppEval separates a broken mobile app from a broken testing environment. A task qualifies only when the same installed-app target reaches a failing behavioral assertion in the defective version and passes after the reference fix. Missing development tools, unavailable devices and other infrastructure failures remain separate outcomes. This makes the benchmark's comparison depend on exercising the app's behavior.
The authors report first-attempt success rates ranging from 22% to 90.5% across five agents on 200 Android tasks from 24 repositories. Although the framework also covers iOS and HarmonyOS, those quantitative results apply specifically to Android.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.