SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References
Source
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
Author
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
If you are generating summaries to feed an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, this says optimize for correctness and the right abstraction level and stop paying for polish the model does not use.
Key quotes
“But this overlooks a growing reality: LLM-based tools and agents increasingly consume code summaries as inputs for software engineering (SE) tasks, and what makes a summary useful for a consuming agent on a task remains largely unexplored.”
“We find that Correctness, followed by Abstraction, significantly correlates with LLM performance, with correlations up to 14X higher than the best baseline.”
“Conciseness and Fluency, though widely valued by human developers, remain mostly insignificant to an LLM consumer, suggesting that what makes a summary useful is task- and consumer-dependent.”
