Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex
Source
AI Engineer
Author
AI Engineer
Date
Key takeaways · AI-distilled
Jerry Liu on why PDFs defeat agents: a PDF stores text as glyphs with coordinates and tables as line segments, and never records reading order for multicolumn pages, so that structure has to be reconstructed.
Liu frames RAG in 2026 as an agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → plus a context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → layer: retrieval complexity moved into agents that reason about the right search term instead of tuning top-k retrieval, while context moved up the stack into MCP servers and skills.
On parsing, he says one-shot VLM reads hallucinationWhen a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.Full definition → on text pages and cost a lot, and pitches hybrid routing between cheap specialized models and frontier ones. His ParseBench scores about 50 models on 2,000 human-verified pages.
He separates three regimes: high accuracy for regulated finance, low cost for indexing a million documents a day, and low latency for a thousand uploads a minute, where LlamaIndex's free Rust parser LightParse runs as a first pass before a VLM.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
Why it matters
Offers AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → builders a specific framework (parsing, semantic storage, repeatable workflows) and a benchmarked way to choose between cheap and frontier parsing models for real document pipelines.
