The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

A browser agent needs to establish what changed after it acts. The chrome-agent project, discussed in this automation talk, exposes Chrome's debugging interface so an agent can locate a page element, send an input event and inspect the resulting page state.

The verification step closes the gap between issuing a click and establishing that it had the intended effect. Subsequent decisions can then use the observed result rather than assume that the previous instruction worked. This observation-action-verification cycle is the useful architectural point behind the demonstrations.

“Preparing this talk got Corey Gallon a warning from OpenAI that his account faced a ban for cyber abuse with a web browser.”

AI Engineer

“a browser driven through the Chrome DevTools Protocol is just a meat bag with a mouse, because the agent's clicks and keystrokes travel the same path inside Chrome that yours do”

AI Engineer

“Chrome stamps every event as trusted or untrusted, which is why a synthetic JavaScript click that works fine in Outlook gets silently dropped by Amazon's add to cart button.”

AI Engineer

“He argues for a CLI over an MCP server on speed rather than capability, citing a study where both cleared tasks about 83% of the time while MCP took 71 round trips and eight minutes against seven turns and under a minute.”

AI Engineer

“His solution splits the work: deterministic code drives the whole challenge and rearms itself, and the agent is called in only to look at the grid and name the tiles.”

AI Engineer