Runway tested whether a video-based simulator could help choose between robot control systems before testing each one on hardware. In its GWM-Robotics study, eight control systems responded to simulated observations, and human graders scored progress across 1,450 generated attempts. The resulting scores correlated at 0.95 with real-world results.
The study supports using simulation to help rank candidate controllers for the tested tabletop tasks. It does not establish that a simulated success rate predicts the exact real-world rate. All evaluated tasks used a single robot-arm type, so the reported result is narrower than a general replacement for physical robot testing.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
