Vibeleaderboard
← All Intel
Intel / article

Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos

Source
developer.nvidia.com
Author
Nakul Arora
Date

Context

NVIDIA's "two prompts" are instructions to a coding that sets up and runs a training-data pipeline for robots, self-driving cars and camera systems. Models that act in the physical world need examples of rare situations, such as a stalled vehicle, a fire or a defective circuit board, that are slow, costly or dangerous to film. Cosmos Data Factory, part of NVIDIA's Cosmos platform for world models, packages the steps that fill those gaps as five ready-made pipelines, which NVIDIA calls blueprints, and six skills that an agent such as a coding assistant can call.

The first prompt tells the agent to clone the CDF Orchestration repository and run its preflight and setup: preflight checks the environment without changing anything, and setup prepares it. The second prompt names a blueprint and the data. Most blueprints chain similar steps. A search tool or a vision-, which describes images and video in text, works out what is in the source footage; a Cosmos generation model produces new variants, such as the same street in rain or fog, or the same scene with a collision; Cosmos Evaluator checks that the requested condition appears and the motion is physically plausible, retrying failures in the rare-event blueprint; and an auto-labeling step adds detections, tracks and captions so the output can go straight into training. The defect blueprint can also start from 3D models of parts rendered in NVIDIA's Isaac Sim.

NVIDIA's cost claims are its own compute-only estimates. It measured each blueprint's daily output on a cloud server with eight RTX PRO 6000 Blackwell GPUs at about $40 an hour, $960 a day, and divided the daily cost by that output: about 130 driving clips a day at $2.66 each, which it puts at about 45 times cheaper than a $125 road-collected scenario, and a drop from two months to one day to assemble a defect dataset. NVIDIA notes that data preparation, setup, agent use, storage and human review are not included. The customer results it lists are mostly framed as "up to" figures, and some are expected or targeted rather than achieved. Apart from one reported gain of up to 24% in recall on real-world anomalies, the post shows what the pipeline produces and what it costs to run rather than how much models trained on the generated data improve.

Key takeaways · AI-distilled
  • NVIDIA Cosmos Data Factory packages five reference blueprints and six composable agent skills, from auto labeling and defect generation to simulation and orchestration, so a coding agent can set up and run a physical AI data pipeline in two prompts.
  • The generation blueprints check their own output: Cosmos Evaluator verifies the requested condition or event appears and scores physical plausibility, and the event blueprint retries failed clips before auto-labeling the ones that pass.
  • NVIDIA's cost math uses an eight-GPU RTX PRO 6000 server at about $960 a day: roughly 130 AV edge-case clips a day at $2.66 each, about 45x cheaper than a $125 on-road scenario. The figures cover compute only, not data prep, storage or human review.
  • The defect blueprint generates defects from clean inspection images or 3D USD assets, ships fine-tuned for circuit boards, glass and metal, and outputs manifests tracing each image to its source. NVIDIA says it cut dataset assembly from two months to one day.
  • Cosmos Dataset Search is now open source under Apache 2.0, and Cosmos Curator added multicamera video, IMU, GPS and initial lidar inputs, according to NVIDIA.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Read the source developer.nvidia.com
Recommended reads
Comments

Checking sign-in…

Loading comments…