- Category
- Developer Tools
- Rank
- No. 1219Tools index
Previous survey · No. 1206 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- Interfaces
- CLI
- GitHub
- 195 stars
- Date
About
An open-source framework from Preference Model for building robust reinforcement learning environments to train aligned AI. It requires Python 3.12+ and uv, and runs environments inside a VM by default (Apple container on macOS, Firecracker on Linux), with Docker or Podman as alternatives.
What it does
Karotte is a Python toolkit for the sandboxes a model works in while it is being trained by trial and reward. You define tasks, the tools the model may call and the judges that score its work; Karotte builds that into an image and runs each attempt in a fresh microVM or container. The model's commands run as an unprivileged user behind a firewall with memory, process and disk limits, and its handed-in files are copied somewhere it cannot touch before any grading starts. Every attempt leaves a transcript you can browse in a terminal dashboard.
Why it's ranked here
The case rests on how specifically it handles reward hacking, the failure where a trained model learns to game the grader instead of doing the task. Its docs list concrete attacks, such as a symlink pointing the grader at the answer key, a pipe that hangs the grader forever, or a planted Python file the grader runs as root, and name the defense for each. MIT licensed, with templates under MIT-0 so the environments you generate need no license notice, installable as a single command-line tool, and released on every merge to main.
What's good
Isolation is per attempt and is a real VM by default: Apple's container on recent Apple silicon Macs, Firecracker on Linux with hardware virtualization, Modal in the cloud. A comparison table states exactly which limits each runtime enforces in the kernel and which only a watchdog catches, rather than implying they are equal. On Linux, Karotte pins the Firecracker and guest kernel downloads to checksums and refuses a mismatch. Judges compose with and/or logic, and a model that tampers with its submission scores zero instead of crashing the run. Model calls retry with backoff and nudge a model that returns an empty turn.
Tradeoffs
Outside a VM the guarantees weaken: under plain Docker or Podman the memory limit is off unless a size is configured, and disk and process limits are only watched, not prevented. Firecracker accepts only read-only mounts and cannot reach a model proxy on the same machine; Ubuntu 24.04 needs an AppArmor change or root to start its network. Modal has no GPUs and lacks several run flags. Collecting a submission deletes every file the model owns, so multi-step tasks must restore earlier work. The language template installs GPL compilers, and you carry the license duty if you ship the image. The model-call library is pinned to a narrow version range.
How to use it well
It fits teams training or evaluating coding agents on tasks with checkable outcomes, where a grader exploit costs real training compute. Run it on a Linux box with KVM or a recent Apple silicon Mac so you get kernel-enforced limits, and run its confinement check inside the sandbox to confirm what you actually got. Always grade the protected copy of a submission, and have scoring scripts turn tampering into a zero. It is not a training loop or a benchmark suite: you still bring the model, the tasks and the optimizer.
Technical notes+
pyproject.toml declares requires-python >=3.12, license "MIT AND MIT-0", a uv_build backend, and the console script karotte = karotte.cli:entry; core dependencies include fastmcp 3.x, litellm pinned >=1.104.0,<1.106 (a comment cites an mcp 2.x import in 1.106), typer, textual and pydantic, with modal and gcs as extras. A [tool.uv] exclude-newer of 7 days acts as a supply-chain age delay. src/karotte/cli/__init__.py wires create-env, run, build, dashboard, update and the agents/check/models/tasks/templates groups, installs a SIGTERM handler that raises KeyboardInterrupt so sandboxes are torn down, and loads plugin commands from the karotte.cli entry-point group without letting them replace built-ins. Its entry function hides the run config from procfs before forking an init shim. src/karotte/__init__.py exports Step, Task, Judge, ProtectedStore, save_submission, kill_processes, delete_files and limit_resources. docs/tasks/scoring.md documents ExecutableJudge, RubricJudge (one LLM call per criterion), RegexJudge and AlwaysPassJudge, combined with & and |, and the five-step collect_submission sequence. docs/running/runtimes.md gives the default sandbox as 2 CPUs and 4 GiB, with the student limited to sandbox memory minus 1 GiB.
Observed
- License
- MIT for the framework, MIT-0 for environment templates (pyproject.toml)
- Language
- Python, requires 3.12 or newer
- Install
- uv tool install karotte
- Interfaces
- CLI (karotte) plus a Python library for tasks, steps, tools and judges
- Default runtime
- Apple container on macOS 26+ Apple silicon, Firecracker on Linux with KVM, Docker elsewhere
- Other runtimes
- Modal, Podman, nerdctl, Docker with gVisor
- Model access
- Calls providers through litellm
- Release policy
- README states every merge to main is released
- Extensibility
- Plugin CLI commands registered through Python entry points
Read from README.md, pyproject.toml, docs/why-karotte.md, docs/getting-started/concepts.md, docs/getting-started/installation.md, docs/running/runtimes.md, docs/tasks/scoring.md, src/karotte/__init__.py, src/karotte/cli/__init__.py, src/karotte/firecracker/__init__.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.