Rebuilding the web for agents — Liad Yosef, MCP Apps
Source
youtube.com
Author
AI Engineer
Date
Context
Publishing instructions for agents does not ensure agents will find them. In the tests described in this MCP Apps talk, roughly half the tested websites supplied an llms.txtA file at a site's root that gives AI systems a curated, plain-text map of its most useful content — robots.txt's counterpart for comprehension rather than permission.Full definition → file, but agents generally visited documentation or the homepage instead. The few that opened the file found it through mentions in the documentation.
The reported observation concerns a discovery path: guidance placed at a conventional address can still be missed if the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s browsing never reaches it. It supports making guidance discoverable from pages agents actually visit, while leaving the broader effectiveness of llms.txt dependent on the agents and sites being tested.
Key takeaways · AI-distilled
Of the websites Yosef's team tested, roughly half publish an llms.txt file, but almost none of the agents they ran against those sites ever read it; agents went straight to the docs page and then the homepage instead.
The minority of agents that did open llms.txt only found it because the site's own docs happened to mention the file existed, not because the agent looked for it directly.
Yosef co-created and maintains the MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → Apps spec, which lets a server hand back a real interface to a chat instead of flattening everything into text; he argues shipping it is what opened the door to an agentic web.
His team asked their own coding agent to pick an analytics service and it recommended a vendor they'd never used over the one they'd run for a decade, purely because the new vendor had a better API to integrate against, and they switched.
Terms in this piece · Glossary
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
llms.txt — A file at a site's root that gives AI systems a curated, plain-text map of its most useful content — robots.txt's counterpart for comprehension rather than permission.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
videoVoice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
video78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble
videoWhen Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
videoHow to Decide What Your AI Agent Should Remember - Wojciech Pluta, Oracle
