LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
Offers a concrete, tested case study in designing technical evaluations that resist LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → saturation, useful for anyone building hiring processes or benchmarks in a world where frontier models increasingly match human experts under time constraints.
Key quotes
“A take-home that distinguishes well between human skill levels today may be trivially solved by models tomorrow—rendering it useless for evaluation.”
“When given the same time limit, Claude Opus 4 outperformed most human applicants.”
“Humans can still outperform models when given unlimited time, but under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model.”
“While the work we do has evolved alongside our models, we still need more strong engineers—just increasingly creative ways to find them.”
