← All IntelClip / Developer ToolsUnrealistic, leaky prompts (SWE-Bench Pro example)
From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈3:37
“That is not how people write prompts.”
“there's 481 words per instruction in average.”
“That's a two-pager per task.”
What’s in it
- Exposes how a popular coding benchmark leaks the answer into prompts
- Shows real examples of 'leaky prompts' that spoon-feed LLMs solutions
- Argues benchmark task instructions look nothing like real developer prompts
Clip transcript
Um but, that's not the reality. So, what what what went wrong? So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic. Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts. And to illustrate more of that, um I took a couple examples here. The first one I looked at I I call the leaky prompt. It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects. So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that. The second one is is even worse. Um it's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way. So, that's the leaky prompt.
Comments
Sign in to comment.
Loading comments…