When Learned Context Planning Fails to Beat Strong Retrieval: A Controlled Study of Planning, Routing, and Reranking for Long-Context QA
Source
arxiv.org
Author
Yingrui Li, Han Chen
Date
Context
A learned context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → planner selects pieces of evidence before the answering model sees them. In this long-document question-answering study, a trained planner did not beat strong retrieval. Using Qwen2.5-7B-Instruct as the answering model, the researchers compared planner-guided selection with methods that retrieve relevant evidence directly.
On 152 untouched test questions, the hybrid retrieval baseline reached 42.11% accuracy, versus 36.84% for a planner-guided method. This held-out result matters because the broader 503-question analysis included questions used during planner development. Within the tested setup, adding learned planning did not earn its place as a replacement for the stronger retrieval method.
Key takeaways · AI-distilled
At an 18k-character budget with Qwen2.5-7B-Instruct, anchored hybrid retrieval scored 36.18% and BM25 35.98% on all 503 LongBench-v2 MCQs, versus 34.19% for the best planner-guided method.
On the untouched 152-question test split the gap widened: anchored hybrid retrieval reached 42.11% against 36.84% for the planner.
The authors caution that the 503-question analysis is partly transductive, since it includes the 168 questions used to train and tune the planner.
Planner-guided rerankingA second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.Full definition → showed a +1.79-point estimate at a 6k budget, but its paired interval crosses zero and it tied the control at 9k; the authors call learned planning a weak relevance signal.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
reranking — A second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.
articleContext Poisoning as Extreme-Value Attention Interference in Long-Context Language ModelsMeysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato
articleMEMENTO: Learning Context Management for Long Context Processing AgentsYunfan Zhang, Taehoon Hwang, Kathleen McKeown, Smaranda Muresan
articleWhen Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic InterpretabilityMayukh Debnath, Adeyinka Adedigba, Rammohan Mallipeddi