Clip / Developer Tools

Not economically valuable tasks

From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈4:52

Highlights that some benchmark tasks are well-formed but test irrelevant, non-representative real-world work.

What’s in it

Highlights that some benchmark tasks are well-formed but test irrelevant, non-representative real-world work.

Clip transcript

The second example, it's the the not economically valuable prompt. Uh this is from Sweet Marathon. And this prompt is well-formed. It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So, I don't know if any of you ever tried to do that, but I don't think it's a good idea. We should not do that.