System One Harness (SOH)
github.com/harnessrouter/systemoneharness- Category
- AI Agents
- Rank
- No. 1240Tools index
Previous survey · No. 1209 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Agent Building
- GitHub
- 206 stars
- Latest release
- v0.4.0
- Date
About
System One Harness is a framework that turns a "System One" decision model into an agent loop: it observes an environment, compiles the environment's finite action space into typed questions for the model, gates each decision by a confidence threshold, executes the chosen action, and logs the full decision trace. It makes one model call per step with no free-form generated actions, and currently supports the Jev model from TypeSafe via OpenRouter or TypeSafe's own API.
What it does
A Python runner for models that answer multiple-choice questions instead of writing text. You declare every move a program, MCP server or web page allows, with each parameter drawn from a fixed list, and the harness asks the model to pick one, checks its probability against a bar set per risk level, carries the move out and repeats. Every run stops with a named status and reason. Right now the only model it can drive is TypeSafe's Jev, so you need an OpenRouter or TypeSafe key and a paid model call on every step.
Why it's ranked here
The model's probabilities work here as a safety control rather than a log line: the harness refuses any move whose shakiest judgment falls below the bar, records a handoff package when it gives up, and publishes its own numbers: five live runs per scenario on three small order tasks, all fifteen reaching the goal for under three hundredths of a cent per run. The README says plainly that this proves nothing about ambiguous state, arithmetic or dates. Apache 2.0, with a stated test suite covering every way a run can end.
What's good
The confidence check uses the weakest probability among the action and its parameters, not their product, so one doubtful pick blocks the move. Thresholds live in configuration: the bundled example asks 0.5 for reads, 0.6 for writes and 0.8 for destructive steps. When the model picks something it was never offered, the harness refuses instead of trusting the provider. MCP tools it cannot convert are listed with a reason instead of dropped quietly. Free text is rejected outright, and browser form values come from named strings you supply, so the model never types into a page. Rate-limit and server errors retry with doubling waits.
Tradeoffs
You are tied to one model family from one vendor, reached through a beta OpenRouter endpoint or TypeSafe directly. The approach only fits work you can enumerate up front: no free-form input, and per the design notes the model handles arithmetic, counting and long irrelevant context poorly, so your code has to do that filtering. Escalation to a reasoning agent is still marked as later work; today a refusal just ends the run with a handoff record. In real-time mode, low-confidence read actions run anyway. The version number differs between the README badge and the package metadata, so check which one you installed.
How to use it well
Start with the built-in order fulfilment demo, which prints each pick with its probability and latency plus a final status, wall time and cost line. Then point it at your own loop with a short action file listing actions, risk levels and thresholds, or at an MCP server after inspecting how its tools convert. It suits high-volume routine choices such as form steps, triage queues or game input, where a few hundred milliseconds per step matters. For tasks that need written output, planning or judgment over ambiguous state, keep a general language-model agent and treat this as the fast layer underneath.
Technical notes+
systemone_harness/controller.py runs observe, compile, encode, decide, gate, execute in one loop and ends with typed reasons (goal_reached, finish_insisted, environment_terminal, max_steps, timeout, no_confident_action, repeated_action, escalation_requested, environment_error, provider_error, cancelled); a refusal streak of 3 ends non-realtime runs, identical action plus state hash ends with repeated_action, and finish needs goal_reached at or above the finish threshold or a second insistence. systemone_harness/gate.py takes min() over action confidence and parameter probabilities, defaults to 0.7 for an unlisted risk, and refuses unoffered actions with threshold 1.0. systemone_harness/provider.py retries 429, 529 and 5xx with backoff starting at 0.5s over 4 retries and picks OpenRouter before TypeSafe from env keys. pyproject.toml says 0.4.0, the README badge 0.3.1, and the user agent 0.1.0.
Observed
- License
- Apache-2.0
- Language
- Python, requires 3.10 or newer
- Core dependencies
- httpx and pyyaml; optional mcp and browser extras (browser-use)
- Interfaces
- s1 CLI (run, tools, serve, bench), Python library, MCP client, UHP server
- Model providers
- Jev via OpenRouter or TypeSafe API; requires OPENROUTER_API_KEY or TYPESAFE_API_KEY
- Install
- Source checkout installed with pip install -e, per the README quickstart
Read from README.md, pyproject.toml, docs/design.md, systemone_harness/controller.py, systemone_harness/gate.py, systemone_harness/provider.py, LICENSE.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.