Transcript
Every AI tool built for lawyers today claims to read your documents. Most of them do it the same way, using a technique called RAG, short for retrieval-augmented generation.
RAG is the default because it is cheap, fast, and well-understood by engineers. It is also, for legal work specifically, the wrong shape. Understanding why is worth a few minutes, because it explains something a lot of lawyers have already felt without being able to name, which is that these tools sound impressive in a demo and then quietly disappoint on a real matter.
What RAG actually does.
RAG chops your documents into small text fragments, typically a few hundred words each. It converts those fragments into mathematical vectors, which are essentially coordinates in a very high-dimensional space where similar meanings sit close together. When you ask a question, the system converts your question into a vector too, looks up the fragments nearest to it, stuffs them into the model's context window, and asks the model to answer based on what it just received.
That design works well when the source material is a customer support knowledge base, or a Wikipedia-style reference corpus, or any body of text where facts are locally stored, meaning you can answer a question by reading one passage in isolation.
Legal documents are not like that. And this is where the cracks start.
The chunking problem.
A contract is not a pile of paragraphs. It is a structured instrument where clause 14.3 modifies the definition in clause 1.2, which is qualified by a schedule thirty pages later, which is itself subject to a side letter that nobody remembered to name consistently. Meaning is distributed across the document. It is also distributed across documents, because the MSA talks to the SOW, which talks to the DPA, which talks to the order form.
When RAG chops all of this into fragments, it severs those links. The indemnity gets retrieved. The carve-out does not. The cap gets retrieved. The supercap carve-out for gross negligence does not. The AI answers confidently, and it answers wrongly, and the confidence is the dangerous part.
The retrieval lottery.
RAG picks fragments by semantic similarity, meaning fragments whose language looks statistically close to the language of your question. This is not the same as legal relevance, and on messy real-world matters the gap is enormous.
Ask about termination rights and you get every paragraph that contains the word termination, including the boilerplate, the recitals, and an unrelated clause in an old NDA that happened to share vocabulary. Meanwhile, the clause that actually controls your answer, which might be titled something bland like "Consequences of Default," may not make the cut at all, because its language does not happen to match yours. There is also the issue of how many relevant results to pull-out and you need to hardcode the number, this means if there are more relevant results than your top-k as its called, you lose material context, and if there are too few, you might include. alot of irrelevant stuff for the synthesiser, see below.
You end up with a retrieval result that is technically responsive and practically useless. Worse, you cannot tell from the outside which is which.
The synthesis problem.
When RAG returns its fragments, the model sees a scrambled collage. It does not reliably know which fragment came from which document, in what order, or with what authority. It cannot tell you that the MSA and the SOW contradict each other, because it does not have a sense of the two documents as separate objects. It has a blended soup of text.
What the model does next is what language models do, which is produce a fluent, plausible answer. On a legal matter, plausible is not the same as correct, and it is very far from defensible.
This is fine if you are building a chatbot. It is not fine if you are closing a transaction, advising on a dispute, or running a diligence exercise where the cost of a missed clause is measured in millions.
A different shape.
There is a better approach, and it is worth naming because a small number of tools are quietly building in this direction. It is sometimes called map-reduce reasoning, borrowed from a classic idea in distributed computing, and the shape of it is very different from RAG.
Instead of chunking, the system reads each document end to end and builds a structured index of what that document actually contains. Its subject matter. Its key clauses. Its obligations, deadlines, parties, risk points. Not a vector. A structured map. That index becomes the system's understanding of what is in the matter.
When you ask a question, the system works in two stages.
First, it consults that index to decide which documents are actually relevant. Not which fragments look similar. Which documents matter. If your question is about change-of-control, the system knows to open the SHA and the key customer contracts, and to leave the employment agreements alone. This step alone elimin