Dream-Cubed treats Minecraft cubes as tokens for 3D world generation

We are excited to share our latest work, together with @nyuniversity: "Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes." Blog: https://t.co/dASGUTAPIY Paper: https://t.co/EQbWkWgztP Code: https://t.co/QQBh2HmF6i Generative AI has made incredible progress in language modeling, far beyond other modalities, where words and tokens offer a natural compositional unit for scalable training. This is similar to Minecraft and many other popular video games, where developers rely on cubes, tiles, and other discrete primitives to build rich, interactive worlds. In this work, we show that using cubes as tokens allows large transformers to do the same. Our contribution is two-fold: 1/ We release Dream-Cubed to the research community, a large-scale dataset of Minecraft worlds designed for generative modeling. Our data comprises tens of billions of carefully-balanced cubes from procedurally generated Minecraft terrain and high-quality human-authored maps (obtained with the authors' consent). 2/ We use our data to train a family of powerful transformers for efficient generation of interactive 3D environments at cube resolution. We show how our models allow players to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable. Using high-quality data, we demonstrate that these models can be successfully trained with different training objectives, including both continuous and discrete diffusion, unlocking targeted inpainting, large-scale outpainting, and user-conditioned generation of infinitely sized worlds with fine-grained block-level control.

Video · View on X

Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes https://t.co/T5jlKsfxnP