We are excited to share our latest work, together with @nyuniversity: "Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes."
Blog: https://t.co/dASGUTAPIY
Paper: https://t.co/EQbWkWgztP
Code: https://t.co/QQBh2HmF6i
Generative AI has made incredible progress in language modeling, far beyond other modalities, where words and tokens offer a natural compositional unit for scalable training. This is similar to Minecraft and many other popular video games, where developers rely on cubes, tiles, and other discrete primitives to build rich, interactive worlds.
In this work, we show that using cubes as tokens allows large transformers to do the same. Our contribution is two-fold:
1/ We release Dream-Cubed to the research community, a large-scale dataset of Minecraft worlds designed for generative modeling. Our data comprises tens of billions of carefully-balanced cubes from procedurally generated Minecraft terrain and high-quality human-authored maps (obtained with the authors' consent).
2/ We use our data to train a family of powerful transformers for efficient generation of interactive 3D environments at cube resolution. We show how our models allow players to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable.
Using high-quality data, we demonstrate that these models can be successfully trained with different training objectives, including both continuous and discrete diffusion, unlocking targeted inpainting, large-scale outpainting, and user-conditioned generation of infinitely sized worlds with fine-grained block-level control.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Discrete cubes work as tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → the way words do for language. The released dataset and transformers generate 3D environments that stay editable and playable, giving world-model work an open training substrate at cube resolution.
articleAutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model ResearchMarjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva, Harold Chaput, Mohammad Reza Taesiri