When Learned Context Planning Fails to Beat Strong Retrieval: A Controlled Study of Planning, Routing, and Reranking for Long-Context QA
Source
Yingrui Li, Han Chen
Author
Yingrui Li, Han Chen
Date
Key takeaways · AI-distilled
At an 18k-character budget with Qwen2.5-7B-Instruct, anchored hybrid retrieval scored 36.18% and BM25 35.98% on all 503 LongBench-v2 MCQs, versus 34.19% for the best planner-guided method.
On the untouched 152-question test split the gap widened: anchored hybrid retrieval reached 42.11% against 36.84% for the planner.
The authors caution that the 503-question analysis is partly transductive, since it includes the 168 questions used to train and tune the planner.
Planner-guided rerankingA second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.Full definition → showed a +1.79-point estimate at a 6k budget, but its paired interval crosses zero and it tied the control at 9k; the authors call learned planning a weak relevance signal.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
reranking — A second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.
Why it matters
Tells engineers building long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → or AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → pipelines that a learned planning/routing layer may not beat a well-tuned retrieval baseline, saving effort spent over-engineering context selection.
articleContext Poisoning as Extreme-Value Attention Interference in Long-Context Language ModelsMeysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato
articleBetter Together: Complementary Query Rewriting Under a Strong RAG BaselineSara Shanian, Xiaoqin Yi, Pavlo Ruban, Kurt MacDonald
articleThe Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code ContextAlexander Adkins, Teimuraz Trapaidze