AppCopilot
https://github.com/openbmb/appcopilot- Category
- AI Agents
- Rank
- No. 1135Tools index
Previous survey · No. 1140 ·
- Pricing
- Open Source
- Type
- AGENT
- Builder
- openbmb
- GitHub
- 245 stars
- Date
About
General-purpose mobile agent driven by multimodal foundation models for long-horizon tasks on phone apps.
What it does
AppCopilot watches an Android screen, asks a vision-capable model for the next action, then performs taps, swipes, key presses, or text entry through ADB. It repeats this loop until completion. It also supports voice input, spoken feedback, OCR, recorded action replay, and coordinated work across two phones.
Why it's ranked here
This is a substantial research prototype with a complete screenshot-to-action loop, not merely a model description. Device control, OCR, task logging, experience replay, action voting, and two-phone coordination are implemented. The demanding setup and hard-coded configuration keep it better suited to experimentation than immediate production use.
What's good
The control layer handles emulators and physical Android devices, scales normalized coordinates to screen resolution, and adds Unicode input through YADB. Task runs retain screenshots, actions, and model responses for inspection or replay. Optional OCR can read the final screen aloud, while cross-device support passes extracted information between two phones.
Tradeoffs
Installation spans Android Studio, ADB, YADB, Python dependencies, downloaded models, and one or more vLLM services. Users must edit model endpoints and credentials in source. OCR expects local model directories, cross-device work needs two connected phones and socket ports, and task execution waits a fixed interval after every action.
How to use it well
Use AppCopilot for Android-agent research, controlled demonstrations, and experiments where step logs, replay, OCR, or two-device workflows matter. Start with an emulator and predefined tasks before adding voice or cross-device coordination. It does not replace model hosting, Android tooling, OCR model provisioning, or production deployment infrastructure.
Technical notes+
run_agent.py provides the argparse CLI and drives a screenshot, multimodal prediction, structured action, ADB execution loop through GUITaskExecutor. adb_utils.py implements device discovery, screenshots, normalized taps and swipes, key events, ASCII input, and YADB-backed Unicode entry. log/log_recorder.py stores JSON action logs plus screenshots, while log/log_replay.py replays recorded actions. user/ocr_service.py runs PaddleOCR in a background thread. cross_device_agent.py coordinates two Android devices over sockets. omni_parser/paser.py can correct predicted points against parsed interface bounding boxes. wrappers/utils.py includes action majority voting. Runtime dependencies are pinned or listed in requirements.txt, while endpoints, ports, credentials, schemas, prompts, and predefined tasks live in wrappers/constants.py.
Observed
- Primary language
- Python
- Install surface
- Source clone plus pip requirements, Android Studio, ADB, YADB, downloaded models, and vLLM services
- Interface
- Command-line task runner with predefined tasks, custom text or voice input, and optional execution features
- Mobile platform
- Android emulators and physical Android devices connected through ADB
- Model service
- OpenAI-compatible HTTP chat-completions endpoints served locally or configured through a base URL
- Cross-device support
- A separate command-line coordinator assigns work across two connected Android devices using sockets
Read from README.md, requirements.txt, adb_utils.py, run_agent.py, cross_device_agent.py, audio/tts.py, log/log_replay.py, wrappers/utils.py, audio/audio_play.py, log/log_recorder.py, user/ocr_service.py, omni_parser/paser.py, user/user_manager.py, wrappers/constants.py.
What it can do
Navigate through mobile app interfaces automatically
User instructions and app screen content → Completed navigation sequences and interactions
Execute multi-step tasks across different mobile apps
Complex user goals and app ecosystem → Completed long-horizon workflows
Interpret and respond to visual app elements
Screenshots and visual interface components → Appropriate touch interactions and gestures
Process natural language commands for mobile automation
Conversational user requests → Translated actions and app operations
Coordinate actions between multiple mobile applications
Cross-app workflow requirements → Seamless inter-app data transfer and operations
Learn and adapt to new mobile app interfaces
Novel app layouts and interaction patterns → Updated behavioral models for app control
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.