Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop
- Source
- AI Engineer
- Author
- AI Engineer
- Date
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
The floor, not the ceiling, is what breaks user trust, and the two-number triage gives teams a way to rank an effectively infinite issue list.
“Build the thousand example eval suite everyone tells you to build, switch harnesses, and 80% of it stops meaning anything.”
AI Engineer
“the useful question is not what issues your agent has, since it will have effectively infinite issues, but which ones matter.”
Ben Hylak
“The floor is what breaks trust: recommending a competitor, deleting data, sending slop to a customer because the agent happened to have email access.”
Ben Hylak
“agents are poor at finding anomalies while being good at investigating them, so surface something deterministic like a keyword spike first and hand them that.”
Ben Hylak
“evals now belong in your repo as tests rather than in a prompt playground, because the harness is the product.”
Ben Hylak
videoDistill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash
videoTeaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify
videoWhy LLM Recommenders Will Be AI's Biggest Consumer App — Devansh Tandon, Meta
videoWorld Models Need Causality, Not Pretty Pixels — Christopher Manning, Moonlake AI
Checking sign-in…
Loading comments…