AutoCua

At a glance

AutoCua is an MIT-licensed Python package that lets an AI model operate your computer: desktop apps on macOS, Windows or Linux, a terminal coding agent, your own Chrome through an extension, and iOS or Android devices. You give one task string and pick a model provider, and the agent clicks and types with real operating-system input until it reports done.

Good fit

Developers who want to script multi-step jobs across ordinary desktop apps, browsers or phones from one Python call, and who are comfortable supplying their own model key. The design picks numbered elements from the accessibility tree plus OCR instead of guessing pixel coordinates, and a compression agent summarizes long transcripts so lengthy tasks are meant to keep running.

Consider something else when

Skip it if you are not prepared to let an agent drive your real mouse, keyboard and signed-in browser profile, or if you need a hosted service with no local setup. On Linux, desktop control is tested only on GNOME with Wayland (the README documents a KDE portal swap), and iPhone or iPad control requires a Mac with full Xcode.

Start here

Run pip install AutoCua, put a provider key in a .env file, then save a main.py that calls run_agent with mode="web use", a provider, a model name copied exactly from model_list.txt, and a task such as summarising a Wikipedia article. Run python main.py. Add ui=True later to open the desktop app instead of the terminal.

Cost and requirements

The software is free under MIT, but every run calls a model provider you pay or configure yourself; the README lists 48 models across 9 providers. It needs Python 3.10 or newer. Web use needs Google Chrome. The Android emulator downloads about 2.5 GB on first run. On Linux, computer use and shell use need a one-time install of distro packages for AT-SPI, GTK and the desktop portal. Prebuilt Rust agents ship for Windows x64, Apple Silicon macOS and Linux x86_64; other platforms build from source and need Rust and a C compiler.

Limits to know

Without the separately installed Interception kernel driver, a Windows task that hits a UAC elevation prompt stops there, and that driver is licensed non-commercial or paid commercial. X11 on Linux is untested. The read-only rule for its scout agents' shell commands is enforced only by the prompt, not structurally. The no-telemetry statement and the speed demos are the project's own claims.

Still unclear

• How often does the agent complete tasks correctly, and what guards stop it from taking a destructive action on your machine or accounts?
• What does a typical task cost in model API usage?