VisionClaw
github.com/intent-lab/visionclaw- Category
- AI Agents
- Rank
- No. 1097Tools index
Previous survey · No. 1084 ·
- Pricing
- Open Source
- Type
- APP
- Use case
- Productivity & Collaboration
- Builder
- @rohanpaul_ai
- GitHub
- 2.6k stars
- Date
About
A real-time AI assistant that works with Meta Ray-Ban smart glasses to see what you see, hear what you say, and take actions on your behalf through voice commands. Combines live vision, voice interaction, and app integration through Gemini Live API.
What it does
This is a wearable companion app that turns a phone or smart glasses camera into an always-on visual feed for a voice assistant. A background service streams low frame-rate video and continuous audio to a hosted multimodal model, which narrates what it sees and answers spoken questions. A separate execution layer takes structured commands from that model and carries them out: sending messages, running web searches, controlling a browser, or reaching into connected calendars and note apps on the user's behalf.
Why it's ranked here
VisionClaw is included for its scope: a working reference implementation that connects a phone or wearable camera to a live multimodal model and a separate action-taking gateway, spanning four codebases and several external services. That scope has a cost. The app itself ships under a custom Developer Terms license rather than a standard open license, and the default hosted backend is a closed pilot for a user study, not something a new user can sign into. Evaluating it seriously means standing up a voice worker, a gateway, and matching credentials across two more cloud services before a single call connects.
What's good
The engineering shows real attention to the failure modes a voice assistant hits in practice. A phone-only mode lets someone try the whole pipeline without owning glasses. The gateway distinguishes a spawn acknowledgment from a final answer, so the assistant speaks a natural filler line instead of stalling silently on a slow web search or account action. A stuck session waiting on an unanswered tool confirmation is detected and cleared automatically rather than wedging a user's account for good. The interaction trace explicitly strips any image-shaped field before anything is written to disk, so a privacy boundary is enforced in code rather than left to policy.
Tradeoffs
The project's own license is not standard open source: it places use of the toolkit under Meta's Developer Terms and Acceptable Use Policy rather than a permissive grant, even though some bundled components carry Apache 2.0 notices. Getting a working build is not simple either. The Android build needs a GitHub personal access token with package-read scope just to pull a public SDK dependency, and full voice mode needs a self-hosted gateway plus a separate LiveKit Cloud project with matching credentials on both sides, or the app waits indefinitely instead of failing. The default hosted gateway is a private pilot instance, not a service the public can register on.
How to use it well
This fits a developer who wants a working example of wiring a phone or wearable camera into a live multimodal model and a separate execution layer, and is comfortable standing up cloud infrastructure to get there: a GitHub token, a Gemini key, and, for full voice mode, a LiveKit project and a self-hosted gateway with matching secrets on each side. It is not a good fit for someone who wants a plug-and-play assistant to install and sign into immediately, since the maintainers' own hosted backend is a private pilot and not open for public sign-in. Treat the phone-camera mode as the fastest path to a first working demo.
Technical notes+
The Python voice worker in agent/main.py runs on LiveKit and lets the user choose between Gemini Live's native audio pipeline and OpenAI's Realtime model, tracked via participant metadata, with a StalenessSampler class logging the gap between frames the model actually sees. The Node gateway (gateway/src/server.ts) exposes an OpenAI-compatible /v1/chat/completions endpoint with both a streaming and a blocking path, and gateway/src/turn.ts drives Anthropic's managed-agent sessions API, tracking tool calls, thinking, and stop reasons in a TurnStats object, with logic in turn.ts that clears a session stuck in requires_action before retrying a turn. Auth in gateway/src/auth.ts supports three paths: static GATEWAY_TOKENS read in config.ts, a signed-state Google OAuth self-registration flow with sha256-hashed tokens capped at five per account in store.ts, and a service-token-plus-header path for the trusted LiveKit worker. gateway/src/provision.ts creates one shared Anthropic agent plus per-user vaults and sessions, reconciling the agent's model, system prompt, and MCP tool roster against config on a ten-minute interval rather than on every call. gateway/src/trace.ts writes a per-user JSONL trace to disk through a forbidden-keys filter that strips any image-shaped field before it is written. agent/requirements.txt and gateway/package.json pin the livekit-agents, google-genai, and Anthropic SDK dependencies that the two services share.
Observed
- License
- Meta Wearables Developer Terms and Acceptable Use Policy, not a standard open-source license
- Language
- Swift and Kotlin mobile clients, a Python voice worker, and a TypeScript/Node gateway
- Packaging
- A monorepo of four subprojects: an iOS Xcode project, an Android Studio project, a Python LiveKit worker, and a Node/Express gateway
- Interfaces
- A mobile app, an OpenAI-compatible HTTP chat-completions endpoint, and WebSocket connections for Gemini Live and LiveKit signaling
- Platform support
- iOS (iPhone) and Android (Pixel, Samsung, and other Android phones), with optional Meta Ray-Ban glasses support and a phone-camera fallback mode
- Third-party licensing
- The NOTICE file lists Apache-2.0-licensed components (flatbuffers, folly) bundled from the underlying DAT toolkit, separate from the app's own Developer Terms license
- Dependency distribution
- The Android build depends on Meta's DAT SDK, distributed through GitHub Packages, which requires an authenticated GitHub token even though the repository is public
Read from README.md, agent/main.py, gateway/src/server.ts, gateway/src/turn.ts, gateway/src/auth.ts, gateway/src/provision.ts, gateway/src/trace.ts, gateway/src/store.ts, gateway/src/config.ts, agent/requirements.txt, gateway/package.json, CONTRIBUTING.md, LICENSE, NOTICE.
What it can do
Analyze visual scenes in real-time
Live camera footage from Meta Ray-Ban glasses → Visual scene description and understanding
Process voice commands
Spoken voice commands through glasses microphone → Executed actions or verbal responses
Stream live video to AI processing
Real-time camera feed at 1fps → Continuous visual analysis via Gemini API
Provide bidirectional audio interaction
User voice input and AI responses → Two-way voice conversation through glasses speakers
Integrate with external applications
Voice commands for app actions → Executed tasks in connected applications
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.