• Vector search ranks by wording, not authority. Ask about termination rights and you get every paragraph containing the word “termination,” while the clause that actually controls the answer, titled something bland like “Consequences of Default,” never surfaces.
• Top-k, the number of fragments retrieved, is a hardcoded guess. Set it low and the controlling clause gets cut; set it high and the model wades through irrelevant text. Nothing in the answer tells you which happened.
• The synthesis step sees a scrambled collage. The model cannot reliably tell which fragment came from which document or with what authority, so it cannot tell you the master agreement and the work order contradict each other — it never held them as two objects.
• The map-reduce alternative reads each document end to end and stores a structured map of it — parties, obligations, caps, deadlines — then answers in two passes: pick the documents that matter, then read those in full. Selection happens per document, not per fragment.

• RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
• chunking — Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.

Why current legal AI fails on real matters, and how map-reduce reasoning beats naive RAG for it. The RAG critique generalizes — read it if your retrieval pipeline is confidently wrong on anything that needs whole-document reasoning.

• postAgentic document extraction: decompose the PDF, then reason over the partsAndrew Ng
• articleAn LLM-as-Judge Won't Save The Product—Fixing Your Process WillEugene Yan
• articleReal AI Agents and Real WorkEthan Mollick

Checking sign-in…

Loading comments…