OmniParser
https://github.com/microsoft/omniparser- Category
- AI Tools
- Rank
- No. 231Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- microsoft
- GitHub
- 25.4k stars
- Latest release
- v.2.0.1
- Date
About
Microsoft's screen-parsing model that turns UI screenshots into structured element data — the perception layer for pure-vision GUI agents.
What it does
OmniParser combines OCR, interactive-region detection, and icon captioning to identify screen content. It removes overlapping regions, assigns labels and normalized coordinates, and returns both an annotated image and a structured element list.
Why it's ranked here
The project offers more than a research checkpoint: it includes reusable parsing code, an interactive demo, evaluation material, and an agent loop. Its value is strongest for experimentation, while manual weight handling and a broad dependency stack make deployment less tidy.
What's good
It preserves recognized text while adding descriptions for non-text icons, giving an agent both visual location and semantic context. Detection can use a local TorchScript weight or fetch it from Hugging Face. The parser also supports CPU fallback outside the demo.
Tradeoffs
Setup requires cloning the repository, creating a Python 3.12 environment, installing many computer-vision and agent dependencies, and obtaining model weights. The demo explicitly selects CUDA, despite lower-level CPU support. Model licensing also varies: earlier Ultralytics-based detectors remain AGPL rather than MIT.
How to use it well
Use it when building or testing screenshot-driven GUI agents that need candidate elements before choosing an action. Start with the interactive demo, then integrate the parser output into an agent loop. OmniParser alone does not decide tasks or execute clicks; OmniTool supplies that adjacent workflow.
Technical notes+
util/omniparser.py exposes Omniparser.parse, accepting a base64 image and returning a labeled image plus parsed content. util/utils.py combines EasyOCR or PaddleOCR, overlap filtering, Florence-2 or BLIP-2 captioning, and detector output; Florence loading uses trust_remote_code=True. util/yolov9.py implements a TorchScript YOLOv9Detector, downloads missing weights through hf_hub_download, preprocesses with letterboxing, decodes three stride levels, and applies batched non-maximum suppression. gradio_demo.py initializes models globally and forces torch.device('cuda'). omnitool/gradio/loop.py calls an OmniParser HTTP /parse/ endpoint before each agent step.
Observed
- License
- Repository badge is MIT; the YOLOv9-based detector and caption models are MIT, while earlier Ultralytics-based detector weights retain AGPL licensing.
- Primary language
- Python
- Installation
- Clone the repository, create a Python 3.12 Conda environment, install requirements.txt, and download model weights from Hugging Face when needed.
- Interfaces
- Python parser class, Gradio demo, Streamlit and Gradio OmniTool frontends, and an HTTP parsing endpoint used by the agent client.
- Model inputs and outputs
- Accepts UI screenshots and produces an annotated image with structured text and icon elements.
- Agent platform
- OmniTool is described as controlling a Windows 11 virtual machine.
Read from README.md, requirements.txt, docs/Evaluation.md, gradio_demo.py, util/utils.py, util/yolov9.py, util/omniparser.py, util/box_annotator.py, eval/ss_pro_gpt4o_omniv2.py, omnitool/gradio/app.py, omnitool/gradio/loop.py, omnitool/gradio/app_new.py, omnitool/gradio/app_streamlit.py.
What it can do
Parse UI screenshots into structured data
UI screenshot image → Structured element data
Extract clickable elements from screen captures
Screen capture image → Identified clickable UI elements with coordinates
Convert visual interface layouts to machine-readable format
Interface screenshot → Machine-readable UI structure data
Identify interactive UI components from images
GUI screenshot → Catalogued interactive components with properties
Generate element maps for GUI automation
Application screenshot → Element map for automation scripts
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.