What if the way we collect human feedback in robotics is quietly losing…

A single overall preference between two robot attempts can hide the actual reason one was better. Stanford SAIL's post, crediting Marcel Torne, Anubha Mahajan, Abhijnya Bhat, and Chelsea Finn, gives the example of one trajectory dropping the cutlery and another dropping the plate, where a single preference vote hides which failure mattered.

The paper has annotators name axes like speed or safety and give pairwise judgments along each one in natural language, training a reward model that scores a trajectory once given a text description of an axis, then optimizing a policy against whatever combination of axes was used in training. The authors report a 38 percentage point improvement over sparse-reward and binary-preference baselines across four real-world and two simulated long-horizon manipulation tasks.