Seahaven
github.com/kiln-ai/seahaven- Category
- Developer Tools
- Rank
- No. 3738Tools index
Previous survey · No. 3564 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation · Models: Train & Run
- GitHub
- 61 stars
- Date
About
Seahaven is a Python framework for building synthetic worlds for AI agents, used for reinforcement learning and evaluation environments. Evals and RL need thousands of isolated runs that start from a known state and are graded on what the agent changed, which production systems cannot provide. Seahaven creates working copies of an agent's tools and handles parallel instances, reproducibility, serving, and change logs.
At a glance
Seahaven is a Python framework for building mock services that an agent can change during an evaluation. It is worth considering if you need to check whether an agent changed the right records and left the rest alone.
Good fit
Teams testing agents that call tools or APIs and need repeatable starting data across runs.
Consider something else when
Skip it if you only grade text answers or need to measure production behavior that a mock cannot reproduce.
Start here
Pick one reversible task, such as updating a synthetic customer record. Define its initial state and the expected final change. Compare the recorded changes with both expectations after a run.
Cost and requirements
MIT-licensed Python framework. You supply the mock tools, tables and starting data. Running an external model may bring separate charges.
Limits to know
Each world instance has its own SQLite state. The result depends on how accurately your mock captures the real service, including error cases. A passing run establishes behavior in the scenario you built.
Sources checked 2026-11-05. Source-based assessment.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.