Computer-use models will agentify the web, not APIs — Dhruv Batra, Yutori

A website's visible state can contain information that its initially loaded page code does not express directly. Yutori's talk illustrates this with an unavailable product whose option becomes disabled through a script, rather than displaying explicit sold-out text.

The browser-agent developer argues that agents need to operate the interfaces existing websites already expose, including sites unlikely to build dedicated software interfaces for them. Its Navigator approach uses screenshots to choose actions, can write JavaScript when useful, and checks the result on screen. The argument supports combining visual feedback with code execution when a task depends on the rendered page.

“He grants the popular claim that agents will drive most of the action on the web, then rejects its usual next step, that the web will meet them with APIs. The head of the distribution might. The long tail, some 200 million active sites where infrastructure changes over decades, will not.”

AI Engineer

“A basketball score is missing from the page that first loads and arrives later as JSON. A product page contains no text reading sold out, only a quantity of zero and a script that grays the option out.”

AI Engineer

“State is calculated and rendered rather than stored, which makes the browser closer to a game engine than a document and makes pixels the source of truth.”

AI Engineer

“He calls this the bitter lesson for web agents: scaffolding built per site does not generalize, and the general solution is the one the web was actually built for.”

AI Engineer

“Their Navigator model runs screenshot in and clicks out, now writes JavaScript when that is quicker, and checks the result on screen. It misses 8 of 300 trajectories on a benchmark he thinks should be retired.”

AI Engineer