- Category
- AI Agents
- Rank
- No. 348Tools index
- Pricing
- Open Source
- Type
- AGENT
- Builder
- bytedance
- GitHub
- 11.4k stars
- Date
About
ByteDance's pioneering native GUI agent — one model interprets screenshots and emits clicks/typing to operate any desktop or web app.
What it does
UI-TARS combines an open multimodal agent model with prompts and Python post-processing for interface automation. The model returns a thought and a structured action. The package then normalizes coordinates, parses the action, and produces PyAutoGUI code for mouse, keyboard, drag, scroll, and typing operations.
Why it's ranked here
The project tackles desktop, browser, phone, and game interaction with one model family, and reports strong results across several grounding and task benchmarks. Its practical value is narrower than the headline model capability: this repository mainly provides deployment guidance, prompt templates, coordinate handling, and action conversion, while complete desktop and browser products live elsewhere.
What's good
The action layer handles several model output formats, converts points and boxes, rescales coordinates while preserving aspect ratio, and rejects extreme image shapes. Separate templates cover desktop, mobile, and grounding-only tasks. The Python package has no declared runtime dependencies, installs through common package tools, and includes unit tests for parsing, coordinate normalization, and PyAutoGUI generation.
Tradeoffs
This is not a turnkey automation application. Local desktop control and browser automation are directed to separate projects. Running the documented model endpoint requires GPU infrastructure and several container settings. The parser ultimately emits executable PyAutoGUI source as a string, so adopters must provide the execution loop, screenshots, permissions, failure handling, and safety controls. The supplied unit coverage exercises only three core parser behaviors.
How to use it well
Use it when building or evaluating a vision-language agent and you need a compact bridge from model responses to normalized GUI actions. Start with the platform-specific prompt, verify coordinate conversion visually, inspect generated commands, and execute them inside a controlled environment. Choose another project if you need a finished desktop client, a complete browser automation framework, or an autonomous runtime with production safeguards.
Technical notes+
codes/ui_tars/prompt.py defines desktop, mobile, and grounding action vocabularies. codes/ui_tars/action_parser.py uses ast.parse to decode action expressions, applies Qwen 2.5 VL coordinate normalization through smart_resize, and has parsing_response_to_pyautogui_code assemble executable PyAutoGUI source. codes/pyproject.toml packages ui-tars with Hatchling, requires Python 3.10 or newer, and declares no runtime dependencies. codes/tests/action_parser_test.py contains three unittest cases, while codes/tests/inference_test.py is a coordinate visualization script rather than an assertion-based test. README_deploy.md documents a Hugging Face endpoint and calls it through an OpenAI-compatible client.
Observed
- License
- Apache License 2.0
- Primary language
- Python
- Installation
- Published Python package installable with pip or uv
- Library interface
- Python API for parsing model actions and generating PyAutoGUI source
- Model API
- Hugging Face Inference Endpoint accessed through an OpenAI-compatible client example
- Platform templates
- Desktop, mobile, and grounding-only prompt templates
- Python support
- Python 3.10 or newer, below Python 4.0
- Test structure
- Unit tests cover action parsing, structured conversion, and PyAutoGUI code generation
Read from README.md, codes/ui_tars/prompt.py, codes/ui_tars/action_parser.py, codes/tests/inference_test.py, codes/tests/action_parser_test.py, LICENSE, README_v1.md, README_deploy.md, README_coordinates.md, codes/uv.lock, codes/makefile, codes/README.md, codes/pyproject.toml, codes/.python-version.
What it can do
Interpret desktop application screenshots
Screenshot image of desktop application → Understanding of UI elements and their functions
Interpret web application screenshots
Screenshot image of web application → Understanding of web UI elements and their functions
Generate mouse clicks on UI elements
Target UI element coordinates and user intent → Mouse click commands at specific screen positions
Generate keyboard typing inputs
Text content and target input fields → Keyboard input commands with specified text
Operate desktop applications autonomously
User instructions and desktop application interface → Sequence of automated interactions to complete tasks
Operate web applications autonomously
User instructions and web application interface → Sequence of automated interactions to complete tasks
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.