Three World Labs papers turn 2D generative priors into 3D reasoning
- Source
- x.com
- Date
Today we are sharing three new research papers, each exploring a new way to generate 3D content by leveraging large-scale generative models and 2D priors. These projects were led by our incredible interns @HaoZhang623 @BDuisterhof @DrTunnels [1/4]
World Tracing predicts full 3D from a single image. It outputs a stack of depth values for each input pixel, peeling the world into layers and predicting them with a diffusion model. This predicts full 3D (even occluded surfaces) while remaining faithful to the image. [2/4] https://t.co/sIm3DdOfhr
Modality Forcing adapts text-to-image models to reason jointly about text, images, and depth. It shows that text-to-image is a scalable pretraining objective for 3D reasoning, and how text-to-RGBD, depth estimation, and depth-to-image can be unified in a single model. [3/4] https://t.co/HfYOdgObNw
Flex4DHuman lifts monocular video into dynamic 4D Gaussians. A video diffusion model is finetuned to generate synchronized multiview videos which are distilled into 4D Gaussians. With this method, a video of a person dancing can be lifted to 4D and composted into a 3D world. https://t.co/WUQnfSEq0k
Context
World Tracing, one of three research projects announced by World Labs, addresses a specific obstacle in turning photographs into 3D scenes: a photograph shows the nearest visible surfaces but hides what lies behind them. The paper's abstract describes predicting an ordered stack of 3D points for each image pixel, running from the visible surface into the occluded parts of the scene.
This layered representation aims to preserve correspondence with the input photograph while supplying geometry beyond its visible surfaces. The authors describe uses in text-driven scene editing and generating video from new viewpoints. The hidden geometry is a model prediction, so the contribution is a way to construct plausible, more complete scenes from limited visual evidence.
Checking sign-in…
Loading comments…