Designing AI-resistant technical evaluations

A hiring test loses value when an AI assistant can produce the same result as the strongest applicants within the allotted time. Anthropic's account of redesigning a performance-engineering assessment describes exactly that problem. This assessment explicitly allowed AI assistance, so the designer needed tasks where a person's decisions still affected the outcome.

The replacement used unfamiliar programming puzzles with a tiny, constrained instruction set. Candidates also had to decide whether and how to build debugging tools. The author reports that colleagues could outperform the model on these puzzles, while acknowledging that the replacement sacrificed some realism. It shifted the assessment toward navigating unfamiliar constraints and choosing useful tools.

“A take-home that distinguishes well between human skill levels today may be trivially solved by models tomorrow—rendering it useless for evaluation.”

“When given the same time limit, Claude Opus 4 outperformed most human applicants.”

“Humans can still outperform models when given unlimited time, but under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model.”

“While the work we do has evolved alongside our models, we still need more strong engineers—just increasingly creative ways to find them.”