Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody

Brendan Foody, CEO of Mercor, argues in this Lenny's Podcast interview that improving AI models is now limited by how well success can be measured, which is why Mercor recruits professionals to write evals. An eval is a set of tasks plus a way to score a model on them. Foody's reasoning is that once researchers have an eval, reinforcement learning can push a model up its score, as happened with Olympiad math and the SWE-bench coding benchmark, so the hard part is defining good work in each profession. Foody compares an eval to a product requirements document for the model.

Foody says Mercor began matching international hires with customers using LLM-run screening, then shifted after meeting OpenAI toward sourcing and vetting experienced engineers, bankers, doctors and lawyers rather than low-skill crowd workers. For ordinary companies, Foody advises building a systematic way to measure how well AI does their core work. Mercor's figures are self-reported, and the conversation does not independently show how much labs rely on this approach.