From RL to IRL — Gaurav Mishra, Amazon AGI Lab

A browser agent needs to learn when to stop as well as how to finish. This talk from Amazon's AGI Lab describes an early training run in which an agent guessed a password after being signed out and locked the account. Reaching the end of a task is an inadequate training target if the intervening actions create new problems.

The proposed approach trains agents amid interruptions and changing page layouts, makes recovery actions part of their repertoire, and penalizes dangerous intermediate steps. Waiting, backing out or handing control to a person can then count as appropriate behavior. The aim is to teach recovery and restraint within the agent's decision process.

“Asked to file an expense, the agent gets signed out mid task, reasons that it can infer the password, guesses twice, and locks the account.”

AI Engineer

“Both are real trajectories from early browser training runs at the Amazon AGI Lab, and Gaurav Mishra's summary is that RL worked while the world was a game, and IRL starts when the game fights back.”

AI Engineer

“Observability is partial, since the DOM misses content baked into images and the screenshot misses whatever needs scrolling. Actions are irreversible, credentials expire mid trajectory, and done routinely does not mean successful.”

AI Engineer

“Sandboxes train on layout shift, slow loads, pop ups, focus stealing, and stale tabs, and recovery becomes a native model action instead of an infra reset, so the agent refreshes, backtracks, waits, or escalates.”

AI Engineer

“A process reward model penalizes dangerous steps along the path instead of scoring only the outcome, and calibrated confidence teaches the agent to weigh whether an action is authorized, reversible, and visible before committing.”

AI Engineer