Leviathan
github.com/elstongun/leviathan- Category
- AI Agents
- Rank
- No. 1092Tools index
Previous survey · No. 1087 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Data, Retrieval & Knowledge
- Builder
- @joshuagunnn
- GitHub
- 885 stars
- Latest release
- v0.3.0
- Date
About
Leviathan is a single static binary that turns records (JSONL, JSON, CSV/TSV, SQLite, or database CLI exports) into a ranked full-text index for AI agents. Instead of reading a whole dataset, an agent receives only the few records that answer its question. The author reports that at 1M records it returns 436 tokens instead of 107,000 and finds the answer 99% of the time.
What it does
Leviathan is a Rust command-line tool that loads records from files, SQLite databases or database export commands into one SQLite file with a full-text search index. An agent then asks in ordinary words, optionally narrowed to one customer, machine or other group, and gets back a handful of short result cards carrying record ids it can cite. A separate, optional store gives agents long-term memory: one claim per write, keyed so a newer value replaces the old one, and recalled within a token budget. It runs as a CLI, as an MCP (Model Context Protocol) server over stdio, or as an HTTP server with REST and OAuth.
Why it's ranked here
The project ships a reproducible benchmark that states its own caveats. On a synthetic maintenance log of one million records, the median answer cost 436 tokens against 107,122 for the best grep strategy, with a relevant record in the top five for 99.0 percent of questions. The docs say plainly that the data is synthetic, covers one dataset shape, and that no model read the output, so the accuracy figures are an upper bound. Add an Apache 2.0 license, unsafe Rust forbidden at the lint level and read-only data tools, and this is a carefully built retrieval layer whose headline numbers still need checking against your own data.
What's good
The defaults lean toward refusing rather than guessing. An ambiguous or unknown group name returns the candidates and a distinct exit code instead of picking one. Every answer reports how many matches it showed out of how many existed, so an agent can tell no match from no data. The memory store refuses writes that look like API tokens, private keys or password assignments, and there is no switch to turn that off. Index builds are atomic, and a failed build never replaces a working index. The remote server opens indexes read-only, compares tokens in constant time and puts per-caller limits on its OAuth endpoints.
Tradeoffs
Retrieval is lexical: BM25 ranking with stemming and no embeddings, so a question that says weeping where the records say leak depends on its other words. The benchmark covers one synthetic dataset where each question targets a single entity, and the docs warn that data without a natural group, or questions spanning the whole dataset, behave differently. Leviathan never connects to databases itself, so piped input needs a field mapping written first. The HTTP server speaks plain HTTP, so anything beyond localhost needs a TLS proxy in front. Latency figures come from one 2018 desktop CPU, and the index runs about 1.8 times the size of its source data.
How to use it well
It fits an agent that has to answer questions about one entity inside a large table, log or export: tickets per customer, maintenance history per machine. Let the init command propose a field mapping, review it like a pull request, then index and run describe to check the counts and filter values. If per-session context matters, prefer the CLI with the bundled skill over MCP, since the MCP tool list costs about 640 tokens every session. Use upsert for nightly changes instead of full rebuilds. It does not do semantic search, and for analysis it offers exact counts, sums and averages rather than arbitrary queries.
Technical notes+
Cargo.toml defines a library (src/lib.rs) and a binary (src/main.rs) on edition 2024 with rust-version 1.88, sets unsafe_code = "forbid", and bundles SQLite through rusqlite. A default remote feature gates tiny_http, ureq, sha2 and base64; docs/REMOTE.md notes that --no-default-features builds a binary with no networking code. src/mcp.rs implements JSON-RPC 2.0 MCP over stdio, advertising protocol versions 2024-11-05 through 2025-11-25; its five data tools carry readOnlyHint, have no SQL passthrough or file access, and the search description appends a one-line dataset summary built from describe. src/main.rs maps memory::Refused to exit 2 and unknown groups to exit 3. docs/CONFIG.md documents the inference heuristics, sampling 2,000 records by default.
Observed
- License
- Apache-2.0
- Language
- Rust, edition 2024, minimum toolchain 1.88
- Install
- cargo install leviathan-index, or prebuilt binaries from GitHub Releases
- Interfaces
- CLI, Rust library, MCP over stdio, Streamable HTTP MCP, REST with an OpenAPI description
- Input formats
- JSONL, JSON, CSV/TSV, SQLite, and stdin piped from database CLIs such as psql or duckdb
- Storage
- One SQLite file per index with an FTS5 full-text index; memory lives in a separate SQLite file
- Ranking
- Lexical BM25 with stemming and configurable boosts; no embedding model
- Memory safety
- unsafe_code set to forbid in Cargo.toml
- Networking
- HTTP server and client sit behind the default remote feature and can be compiled out
- Remote auth
- Bearer token by default, optional single-user OAuth 2.1 with PKCE; no TLS built in
Read from README.md, Cargo.toml, LICENSE, docs/CONFIG.md, docs/MEMORY.md, docs/REMOTE.md, docs/BENCHMARKS.md, src/main.rs, src/lib.rs, src/mcp.rs.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.