Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
Source
youtube.com
Author
AI Engineer
Date
Context
Legal-software company Legora's search account illustrates why storage partitions should reflect how documents are used. Its projects were busy for a period and then largely inactive. Mixing document fragments from active and inactive projects in the same partitions meant queries loaded large amounts of irrelevant data, repeatedly displacing useful data from memory.
The talk, given with turbopuffer, describes making each project its own namespace, a separate storage grouping. That lets inactive projects remain in object storage without sharing the same loaded partition as current work. The explanation ties the improvement to project-level organization and the working data kept in memory.
Key takeaways · AI-distilled
Legora's search latency once degraded from a 100ms P99 to 20 seconds because sharding document chunks across ~4,000 partitions put dormant 'dead' legal projects in the same partitions as active ones, so queries kept pulling huge partitions into memory and thrashing the cache.
The fix was making each project its own storage namespace, so an idle project's data rests in object storage at near-zero cost, and each namespace can carry its own encryption key — which is what banks and large firms mean by physical data isolation and key-revocation ability.
In one experiment, the team disabled their SSD cache entirely (instead of building encryption into it), measured the performance hit, found it acceptable, and left the cache off.
The speakers say they favor tree-based clustering over graph-based navigation for object storage, and argue full-text search at web scale actually costs more to run than vector search.
videoVoice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
video78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble
videoWhen Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
videoHow to Decide What Your AI Agent Should Remember - Wojciech Pluta, Oracle
videoGrep or Embeddings? Agentic Search Over Company Documents — George He, LlamaIndexAI Engineer
videoRetrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Eskildsen of TurbopufferLatent Space
videoTurning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, SuperlinkedAI Engineer