← All IntelClip / Developer ToolsBenchmark prompts engineers would never write
From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈1:10
“Nah, it's impossible. No one writes prompts like these ever.”
“Would you ever write a prompt like this?”
“And the answer was no. And they're right. They shouldn't.”
What’s in it
- Exposes how odd real AI benchmark prompts can look
- Reveals engineers reject prompts benchmarks actually use
- Sparks doubt about whether benchmarks reflect real work
Clip transcript
I want to grab your attention. And I invite you to look at this. These beautiful three screenshots are a single prompt on one of the benchmark tasks. As I was looking at it, I was like how can an engineer write a task like this? So I said, "Nah, it's impossible. No one writes prompts like these ever." But I wanted to double-check with my engineers. So I took three of our best engineers. I showed them the prompt and I said, "Would you ever write a prompt like this?" And the answer was no. And they're right. They shouldn't. And so
Comments
Sign in to comment.
Loading comments…