Peekaboo
github.com/steipete/peekaboo- Category
- AI Agents
- Rank
- No. 294Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- @steipete
- GitHub
- 5.1k stars
- Latest release
- v4.3.2
- Date
About
A macOS CLI and MCP server that enables AI agents to capture screenshots and automate GUI interactions through natural language commands. It provides pixel-accurate screen capture, menu discovery, and complete UI automation with support for multiple AI providers including GPT, Claude, and local models.
What it does
Peekaboo turns the Mac desktop into an observe, identify, and act loop. It reads accessibility metadata, assigns element IDs, captures the relevant display or window, then targets controls by label, ID, or explicit coordinates. You can drive that loop from shell commands, a menu-bar app, a built-in planning agent, or an MCP client.
Why it's ranked here
The strongest case is its shared automation core across several useful surfaces. Shell users can script individual actions, while agents and MCP clients reuse the same primitives. Structured UI maps, JSON output, snapshot-bound coordinates, focus checks, and background delivery show serious attention to reliable desktop control. The narrow macOS requirement and substantial permission setup keep it specialized.
What's good
Semantic targeting reduces dependence on fragile screen coordinates. Exact-window snapshots preserve window identity for later actions, while coordinate context maps resized images back to logical display space. Background delivery can avoid stealing focus for supported interactions. The tool also exposes capture, accessibility inspection, menus, windows, dialogs, Spaces, and resumable agent sessions through a consistent command surface.
Tradeoffs
Released builds require macOS 15 or later, and source builds require Swift 6.2 or later. Screen capture needs Screen Recording permission, while inspection and control need Accessibility permission; synthetic input may require another grant. MCP supports standard input and output, but its HTTP and server-sent event transport flags are not implemented. Raw key chords and pointer-heavy actions often require foreground operation.
How to use it well
Use Peekaboo for Mac-only desktop testing, repeatable app workflows, support diagnostics, or agent tasks that must cross native application boundaries. Start by observing an exact window, retain its snapshot reference, prefer accessible labels or element IDs, and use coordinates only with the matching capture receipt. It does not cover Windows or Linux directly, and it is not a substitute for those platforms' separate community rewrites.
Technical notes+
README.md defines the observe-then-act workflow and the CLI, app, agent, and MCP surfaces. package.json publishes Darwin arm64 and x64 binaries through npm, requires Node.js 22+, and exposes both peekaboo and peekaboo-mcp executables. docs/MCP.md specifies a stdio JSON-RPC server, strict numeric validation, inline image responses, versioned coordinate_context, exact-window ROI handling, and snapshot-bound coordinate dispatch. docs/daemon.md describes a reusable CLI daemon with a separate Unix socket from the app Bridge, while MCP remains client-owned over stdio. docs/engine.md documents automatic selection between CoreGraphics and bounded ScreenCaptureKit capture, including process-ownership safeguards. Apps/CLI and Core/PeekabooCore provide the Swift command and service layers described in docs/spec.md.
Observed
- License
- MIT
- Primary language
- Swift
- Platform support
- macOS 15 or later; npm package targets Darwin on arm64 and x64
- Install surfaces
- Homebrew CLI, npm package, signed DMG, and source build
- Interfaces
- CLI, menu-bar app, Swift package, built-in agent, and MCP server
- MCP transport
- stdio is supported and default; HTTP and SSE server transports are not implemented
- Runtime requirement
- The npm package requires Node.js 22 or later
- Testing structure
- Package scripts include safe Swift tests, automation tests, release checks, and Node contract tests
Read from README.md, package.json, docs/MCP.md, docs/tui.md, docs/TODO.md, docs/spec.md, docs/index.md, docs/audio.md, docs/focus.md, docs/oauth.md, docs/README.md, docs/daemon.md, docs/engine.md, docs/install.md, docs/building.md.
What it can do
Capture screenshots of macOS screens
Screen selection or coordinates → High-fidelity screenshot images
Automate GUI interactions through natural language
Natural language commands describing desired actions → Executed GUI interactions (clicks, typing, etc.)
Discover and analyze menu structures
Application menus or UI elements → Menu hierarchy and available options
Analyze screen content with AI
Screenshot images → AI analysis and interpretation of UI elements
Automate workflows across multiple screens
Multi-screen setup and automation commands → Coordinated actions across multiple displays
Execute pixel-accurate screen interactions
Precise pixel coordinates and interaction type → Exact click or interaction at specified location
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.