Vibeleaderboard
← All Intel
Intel / article

The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context

Source
arxiv.org
Author
Alexander Adkins, Teimuraz Trapaidze
Date

Context

Finding more relevant files can still leave a coding assistant with less useful . In The Recall Trap, researchers gave an assistant twelve code excerpts and no search tools. Limiting the pack to one excerpt per file increased how often it contained a file associated with the correct fix, but reduced the detail available from each file. Study abstract.

Removing that restriction raised OpenAI's GPT-5.6 Sol issue-resolution rate from 39.2% to 46.8% across 500 tasks. The finding is specific to how limited context was packed: the effect reversed with a different, lexical retrieval method and was not detected when agents could read freely. Retrieval quality therefore needs to be checked against completed repairs, because file coverage alone can reward the wrong tradeoff. Reported findings.

Terms in this piece · Glossary
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Recommended reads
Comments

Checking sign-in…

Loading comments…