MolmoMotion forecasts object motion in 3D from a few frames and an instruction
- Source
- x.com
- Date
We're releasing MolmoMotion, a 3D motion forecasting model. Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 🧵
MolmoMotion can predict different motions across scenes, like a bowl sliding and rotating on a table, a flamingo dipping its beak as it walks, & a lint roller working back and forth on cloth. Each predicted path follows the instruction and stays close to the ground-truth motion.
MolmoMotion represents motion as 3D points attached to an object, tracked in a shared world frame. This approach doesn't need templates for objects, stays stable as camera perspectives change, & is compact enough to feed straight into downstream applications.
Motion forecasters like MolmoMotion have many potential applications. Fine-tuned, MolmoMotion can predict object paths to help grasping robots plan where to move objects, or help image generators capture motion more accurately—particularly motions hard to describe in a prompt.
Context
Ai2's MolmoMotion predicts paths that points on an object will follow in three-dimensional space. It takes one or a few video frames, the points' initial 3D locations, and a written action instruction. The output describes the object's expected movement over the next few seconds. These paths use a shared world coordinate system, so their meaning stays consistent across camera viewpoints. Ai2's technical explanation describes this compact representation as an input that robot-planning and video-generation systems can use when they need an explicit description of motion.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…