Clip / Developer Tools

Benchmark prompts engineers would never write

From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈1:10

“Nah, it's impossible. No one writes prompts like these ever.”

“Would you ever write a prompt like this?”

“And the answer was no. And they're right. They shouldn't.”

What’s in it

Exposes how odd real AI benchmark prompts can look
Reveals engineers reject prompts benchmarks actually use
Sparks doubt about whether benchmarks reflect real work

Watch this moment

Clip transcript

I want to grab your attention. And I invite you to look at this. These beautiful three screenshots are a single prompt on one of the benchmark tasks. As I was looking at it, I was like how can an engineer write a task like this? So I said, "Nah, it's impossible. No one writes prompts like these ever." But I wanted to double-check with my engineers. So I took three of our best engineers. I showed them the prompt and I said, "Would you ever write a prompt like this?" And the answer was no. And they're right. They shouldn't. And so