Intel / article

How to Use AI Agents to Prepare 3D Scenes for Simulation

Source: developer.nvidia.com
Author: Tanya Lenz
Date: Published Sep 16, 2026

Context

NVIDIA's scene-preparation workflow addresses a gap between a convincing 3D image and a usable robot-training environment. A visible box still needs information describing whether it moves, how it collides and whether it is a target for the robot. The technical walkthrough assigns specialized agents to inspect a Blender scene and add that information to a shared OpenUSD scene representation. Physics tools author properties such as mass and friction; SimReady validation checks the result against a chosen simulation profile. Safe, mechanical problems can be repaired automatically, while decisions that depend on intent, such as an uncertain object label, are escalated to a human. Completion is judged against explicit simulation requirements, rather than the appearance of the exported scene.

Key takeaways · AI-distilled

Codex or Claude coordinates while NVIDIA NemoClaw deploys specialized Hermes subagents (inspection, labeling, materials, sensors, ovphysx, ovrtx, validation), each owning one job with its own acceptance criteria and escalating ambiguous decisions to a human.
OpenUSD is the shared contract layer because it supports layered, nondestructive composition: agents can add labels, physics metadata, and sensor definitions without flattening the original Blender artist's work, unlike fragile one-off exports.
The inspection connects to Blender via a server to inventory the scene (142 objects, 37 materials in the demo) and outputs a structured 'missing' list, such as semantic_labels and collision_meshes, that seeds the rest of the pipeline.
SimReady validation is the hard acceptance gate before Isaac Sim/Isaac Lab handoff: in the demo it reported 14 issues, 10 auto-fixable and 4 requiring review, and only the auto-fixable ones get repaired without a human decision.
Rendering with ovrtx catches problems structural validation misses: it can show a labeled target invisible from a robot-camera view due to occlusion or clipping, prompting Codex to route a follow-up task to sensor or scene-inspection subagents before revalidating.

Terms in this piece · Glossary

multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.

Read the source developer.nvidia.com