openRocode
github.com/wrencanfly/openrocode- Category
- Developer Tools
- Rank
- No. 4088Tools index
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Agent Building
- GitHub
- 1 stars
- Date
About
openRocode is an open agent harness that lets multimodal models control robots the way they use computers. The model looks at camera pictures, points at where the gripper should go, or writes short Python programs, while the harness handles depth, calibration and geometry, applies safety checks to every motion, lets a person intervene, and records each episode. It runs in the MuJoCo simulator and works with Claude or any OpenAI-compatible endpoint.
At a glance
openRocode is an Apache-2.0 Python harness that lets a multimodal model control a simulated robot gripper the way it would use a computer. The model sees labelled camera pictures, clicks a pixel to move the gripper there, nudges in directions it can see, or writes a short Python program for loops and checks. The harness handles depth and geometry, so the model never handles world coordinates. It is alpha and runs only in a MuJoCo tabletop simulation today.
Good fit
Researchers and tinkerers who want to watch how Claude or an OpenAI-compatible model handles pick-and-place tasks, compare models on the same task, or study code-as-policy control. The local dashboard replays every episode turn by turn with both camera views, telemetry and the programs the model wrote, and lets you message, pause or stop a run while it is going.
Consider something else when
You need to control a physical robot or a real arm. The gripper floats, with no arm kinematics, reachability or joint limits, and real-robot adapters are on the roadmap rather than shipped. Skip it too if you need success rates: there is no batch evaluation yet, and one run is one episode.
Start here
Clone the repository, create a Python 3.11+ virtual environment and run pip install -e '.[sim,agent,openai,dev]'. Run rocode demo, a scripted episode that needs no API key, to confirm the install. Then set ANTHROPIC_API_KEY and run rocode run mug_to_coaster --view to watch a model work in the dashboard. rocode tasks lists the other tasks.
Cost and requirements
The software is free under Apache-2.0. It needs Python 3.11 or later, plus MuJoCo, NumPy and Pillow for the simulator. Model runs call either the Anthropic API or any OpenAI-compatible endpoint, such as OpenAI, Qwen, DeepSeek, a vLLM server or a LiteLLM proxy (set OPENAI_API_KEY and OPENAI_BASE_URL, then pass --provider openai). Hosted APIs bill the tokens each episode uses; a model you serve yourself through vLLM carries no per-token fee, only the cost of the machine running it. The demo and the test suite run without a key.
Limits to know
The README says run_code runs model-written programs in a separate interpreter with call and time budgets but is not a security sandbox, so treat it as executing model code on your machine. Only one body exists, a MuJoCo tabletop. Published results are single successful runs on a few tasks, explicitly not success rates. The dashboard listens on 127.0.0.1 only.
Related work
Still unclear
- How often do models succeed across repeated runs? The project reports only single successful episodes until its planned batch evaluation lands.
- How much does a typical episode use in tokens? The dashboard records tokens per run, but the README publishes no figures.
Sources checked 2026-11-18. Source-based assessment.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.