Talking to Itself While Coding: What Makes Comments Help Code Generation?

Comments helped code generation when they supplied useful solution content. In this study, researchers gave weaker models comment blocks taken from stronger models' solutions before asking them to generate code. Comments from solutions that passed the tests improved first-attempt success; comments from failed solutions provided no reliable gain, and comments for a different problem made performance worse.

The researchers also found that comment frequency and broad comment purpose did not reliably predict success. Asking a model to produce comments therefore differs from providing it with a correct plan: the benefit depended on information the comments carried, which the tested prompting approaches generally failed to reproduce.

“Comments from source solutions that pass the tests raise recipient pass@1 by 17.2% on average.”

“comments describing failed solutions provide no reliable gain, while comments written for a different problem reduce pass@1 by 20.8%”

“most recipient models show no significant recovery of the external-comment gain, and the best case recovers only 24%”

“These results show that comments help code generation not merely because they are comments, but because they can provide correct solution content that prompting cannot reliably elicit.”