GR00T-Dreams
https://github.com/nvidia/gr00t-dreams- Category
- AI Tools
- Rank
- No. 900Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 607 stars
- Date
About
NVIDIA GEAR Lab's DreamGen initiative tackling the robotics data problem using world models for embodied AI training.
What it does
GR00T-Dreams turns an image and task instruction into robot videos, estimates actions from those videos, converts the result into LeRobot data, and fine-tunes a robot policy. It also evaluates generated videos for instruction following and physical plausibility.
Why it's ranked here
The project covers an unusually complete research loop: generation, embodiment-specific preprocessing, action recovery, policy training, inference, and evaluation. Its value lies in connecting these stages with working Python tooling. However, critical world-model setup lives in another repository, and the benchmark authors explicitly limit confidence outside its target scenarios.
What's good
The pipeline supports Franka, GR1, SO-100, and RoboCasa workflows. Researchers can train a custom inverse dynamics model from a small set of ground-truth trajectories for another embodiment. The evaluation tools offer both an open vision-language model and an external model option, while deterministic seed handling improves repeatability.
Tradeoffs
This is a heavyweight research stack with many tightly pinned machine-learning dependencies and training settings built around GPU features such as bfloat16 and TF32. Adding an embodiment requires metadata and data-configuration work. The benchmark uses roughly fifty videos per dataset and may generalize poorly to multi-view footage or detailed physics judgments.
How to use it well
Use it when you already operate a robotics training stack and need synthetic videos converted into action-labeled LeRobot trajectories for policy fine-tuning. Start with one of the four supplied embodiments before adding custom metadata. It does not replace the separately configured Cosmos environment or provide a lightweight, turnkey robotics application.
Technical notes+
pyproject.toml packages gr00t with setuptools for Python 3.10 or newer and declares a large pinned dependency surface, including PyTorch, TensorFlow, Transformers, Ray, Diffusers, and platform-specific Decord variants. scripts/idm_training.py and scripts/gr00t_finetune.py expose Tyro-based training CLIs with single-GPU and torchrun multi-GPU paths. scripts/inference_service.py provides ZeroMQ-backed client and server modes through RobotInferenceClient and RobotInferenceServer. IDM_dump/raw_to_lerobot.py writes chunked Parquet episodes and copies MP4 views into LeRobot layout, while IDM_dump/dump_idm_actions.py performs GPU-parallel action inference. dreamgenbench/eval_qwen_pa.py loads Qwen2.5-VL and writes binary physics judgments to CSV, but imports qwen_vl_utils, which is not declared in pyproject.toml. The Makefile defines formatting, linting, pytest, and package-build targets.
Observed
- License
- Apache License 2.0 headers appear in the supplied Python package and scripts.
- Primary language
- Python
- Packaging
- Setuptools project configured through pyproject.toml, requiring Python 3.10 or newer.
- Install surface
- A Python package named gr00t with pinned runtime dependencies and a separate development dependency group.
- Interfaces
- Python library modules, command-line training and evaluation scripts, plus robot inference client and server modes.
- Platform support
- Video decoding selects eva-decord on Darwin and decord on other operating systems.
- Data interchange
- Generated actions and metadata are organized in LeRobot-style directories with Parquet episodes and MP4 videos.
Read from README.md, Makefile, pyproject.toml, gr00t/__init__.py, dreamgenbench/utils.py, scripts/eval_policy.py, scripts/idm_training.py, scripts/load_dataset.py, scripts/gr00t_finetune.py, IDM_dump/raw_to_lerobot.py, IDM_dump/dump_idm_actions.py, IDM_dump/preprocess_video.py, scripts/inference_service.py, IDM_dump/convert_directory.py, dreamgenbench/eval_qwen_pa.py.
What it can do
Generate synthetic robotics training data
Robot task specifications and environment parameters → Synthetic datasets for robot training
Create world models for robotic environments
Environment specifications and physics parameters → Simulated world models
Train embodied AI agents
Synthetic training data and robot behavioral objectives → Trained AI models for robotic control
Simulate robot interactions in virtual environments
Robot models and environment configurations → Simulation results and interaction data
Generate diverse robotic scenarios
Base environment templates and variation parameters → Multiple training scenarios and edge cases
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.