SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

SONAR evaluates code summaries according to their usefulness to another AI system doing software work. Its research abstract describes using a summary to regenerate code, then treating that reconstruction as evidence about summary quality. This avoids requiring a developer-written reference summary against which every candidate must be compared.

Across four downstream software-engineering tasks, the authors found that their correctness measure correlated with the consuming model's performance, while fluency and conciseness were mostly insignificant. The result suggests that readable, economical prose is insufficient evidence that a summary preserves what another coding system needs. The reported relationship is a correlation within those tests, rather than proof that any particular rewriting technique improves performance.

“But this overlooks a growing reality: LLM-based tools and agents increasingly consume code summaries as inputs for software engineering (SE) tasks, and what makes a summary useful for a consuming agent on a task remains largely unexplored.”

“We find that Correctness, followed by Abstraction, significantly correlates with LLM performance, with correlations up to 14X higher than the best baseline.”

“Conciseness and Fluency, though widely valued by human developers, remain mostly insignificant to an LLM consumer, suggesting that what makes a summary useful is task- and consumer-dependent.”