NVIDIA Cosmos
github.com/nvidia/cosmos- Category
- AI Tools
- Rank
- No. 170Tools index
- Type
- APP
- Builder
- nvidia
- GitHub
- 11.8k stars
- Latest release
- Cosmos3
- Date
About
An open platform of world-foundation models, datasets, and tools for building physical-AI and robotics applications.
What it does
Cosmos 3 combines two model behaviors. A reasoner turns text and visual input into explanations, plans, forecasts, and grounded decisions. A generator turns text, images, video, sound, or actions into images, video, synchronized audio, and action trajectories. Both use one Mixture-of-Transformers architecture, switching between causal reasoning and diffusion-based generation.
Why it's ranked here
Cosmos is compelling when one project needs perception, simulation, and robot-action modeling under a shared architecture. The documented integrations span Python experimentation, OpenAI-compatible serving, optimized inference, and containers. That breadth carries a substantial constraint: supported generation targets Linux and specific NVIDIA GPU generations, with larger models aimed at costly data-center hardware.
What's good
The model family covers data-center, workstation, and edge roles instead of forcing one deployment shape. Inputs and outputs have explicit formats, resolutions, frame rates, action dimensions, and hardware guidance. Generator examples include forward dynamics and action policies, while the reasoner covers grounding, planning, and physical plausibility. Evaluation tooling supports dataset groups, retries, result reuse, and parallel execution.
Tradeoffs
Deployment is tightly bounded by the documented stack: Linux, BF16 testing, and NVIDIA Ampere, Hopper, or Blackwell GPUs. The largest model recommends high-end data-center hardware. Edge generation supports fewer resolutions and a narrower frame range, and lacks video-to-video transfer. Sound is generated only alongside video. Post-training recipes and task-specific evaluation are marked as forthcoming. Some supplied evaluation files also carry proprietary confidentiality notices.
How to use it well
Use Cosmos for robotics or autonomous-system teams that need visual reasoning, synthetic scenes, future-state rollouts, or learned action trajectories. Prototype through Diffusers or Transformers, then move compatible workloads to optimized serving or containers. Select the edge model for real-time policies and visual reasoning, and larger variants for synthetic data or distillation. Do not choose it when Linux and supported NVIDIA hardware are unavailable.
Technical notes+
README.md defines Reasoner and Generator surfaces plus Diffusers, Transformers, vLLM, vLLM-Omni, TensorRT-LLM, SGLang, and NIM integration paths. evaluation/cosmos3/reasoner/vlmevalkit/run.py implements CLI-driven model and dataset evaluation with distributed GPU environment handling, inference-only and evaluation-only modes, retries, reuse, and JSON result output. evaluation/cosmos3/reasoner/vlmevalkit/setup.py packages vlmeval for Python 3.7+ and exposes the vlmutil console script. evaluation/cosmos3/generator/unigenbench/utils.py wraps an OpenAI-compatible streaming endpoint with asyncio concurrency, timeouts, and retries. evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py shards video scoring across selected GPUs and uses manifests plus atomic checkpoint writes for resumability. evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/__init__.py globally disables default SSL certificate verification before loading environment values, which deserves security scrutiny. cookbooks/cosmos3/generator/action/finetune/data_processing_for_egocentric_hand_action.py depends on the separate cosmos-framework package for action processing.
Observed
- Implementation language
- The supplied implementation, cookbook, packaging, and evaluation sources are Python.
- Install surface
- Setuptools packaging declares Python 3.7 or newer and installs a vlmutil console command.
- Interfaces
- Python libraries, command-line evaluation tools, OpenAI-compatible serving, and NIM containers are documented.
- Platform support
- Generation is documented for Linux on NVIDIA Ampere, Hopper, and Blackwell GPUs, with BF16 tested.
- Model surfaces
- Reasoner accepts text and vision and returns text; Generator handles vision, sound, and action generation.
- Licensing signals
- Supplied cookbook files identify OpenMDW-1.1, while supplied generator evaluation files carry NVIDIA proprietary confidentiality notices.
Read from README.md, evaluation/cosmos3/reasoner/vlmevalkit/run.py, evaluation/cosmos3/reasoner/vlmevalkit/setup.py, evaluation/cosmos3/generator/unigenbench/utils.py, evaluation/cosmos3/reasoner/vlmevalkit/run_api.py, evaluation/cosmos3/generator/unigenbench/ugb_scorer.py, cookbooks/cosmos3/generator/transfer/preview_helpers.py, evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py, evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/__init__.py, cookbooks/cosmos3/generator/action/finetune/data_processing_for_egocentric_hand_action.py, evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/__init__.py.
Intel on NVIDIA Cosmos
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.