Building Legal Agents When You Can't See the Data - Jacob Lauritzen, CTO of Legora

Legora, which sells an AI agent that lawyers use to draft documents, review contracts and run due diligence, cannot read its agent's traces, the step-by-step records of what the agent did, because the documents belong to its law-firm customers' clients. In a Latent Space podcast episode with co-host swyx and host Basil Chatha of Forward Deployed, CTO Jacob Lauritzen discusses how the company builds and tests the agent anyway. Only the episode page's summary and chapter list were available; the page links a transcript file whose text was not read, so what follows comes from that outline rather than from his answers.

The outline points to two ways of working without customer data. Three engineers and three lawyers build test environments modeled on customers' work, which the summary compares to the simulated worlds used in reinforcement learning, and one chapter mentions testing in a 10,000-file data room. A chapter on product metrics when the traces can't be read is followed by one titled "Paraphrasing traces and narrowing failures down to 10 or 20 types", though the page does not explain how that works. The summary also describes a citation task the model got right 70% of the time that reached 99% after one release, with no change on Legora's part. The chapter that covers it is titled "When evals didn't seem worth it and one release took citations from 70% to 99%", and without the transcript it is unclear what lesson Lauritzen draws, whose release this was, or how citations were scored.