Intel / article

NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation

Source: arxiv.org
Author: Johnny Greco, Nabin Mulepati, Andre Manoel, Eric Tramel, Kirit Thadaka, Mike Knepper, Dhruv Nathawani, Dane Corneil, Yev Meyer, Alex Watson, Maarten Van Segbroeck
Date: Published Sep 17, 2026

Context

NVIDIA released NeMo Data Designer, an open-source framework for generating multi-modal synthetic datasets (arxiv.org). Instead of a bespoke script per project, a user defines each column of the target dataset, spanning text, code, structured outputs, images, , and statistical samplers used to steer how varied the data is, in a declarative configuration file that can be inspected and shared for . Because synthetic data generation is inherently iterative, the framework builds in a preview-and-revision loop: generate a small batch, inspect it, adjust the column specification, then rerun at full scale, while the framework handles dependency resolution, scheduling calls to model endpoints, and retrying failed requests. The authors describe using it internally for Nemotron model development and cite case studies spanning structured, agentic, , and domain-specialized datasets in production enterprise deployments.

Key takeaways · AI-distilled

Dataset columns can be defined with statistical samplers explicitly configured to steer dataset diversity, alongside text, code, structured-output, image, and embedding column types — diversity control is a first-class part of the configuration, not an afterthought.
The full dataset specification is an inspectable, shareable config artifact, and at runtime NDD resolves column dependencies, schedules calls to user-provided model endpoints, and automatically retries failed requests.
NDD was used to build datasets for NVIDIA's own Nemotron model development as well as production enterprise deployments, not just as a research prototype.

Terms in this piece · Glossary

embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.

Read the source arxiv.org