Pi Terminal Bench
https://github.com/badlogic/pi-terminal-bench- Category
- Developer Tools
- Rank
- No. 2046Tools index
Previous survey · No. 2041 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- badlogic
- GitHub
- 32 stars
- Date
About
Harbor agent adapter that lets the pi coding agent run Terminal-Bench evaluations.
What it does
Pi Terminal Bench prepares the pi coding agent inside an evaluation environment, runs it headlessly against terminal tasks, records structured session output, and reports token usage, cache activity, cost, and task results through Harbor.
Why it's ranked here
This is a focused integration with useful accounting and provider flexibility, but its readiness depends on a manual Harbor patch. The included investigation explains that failure precisely, yet requiring users to modify an installed dependency weakens repeatability.
What's good
It preserves structured JSONL output and converts assistant events into input, output, cache, and cost totals. It accepts credentials for several model providers, quotes task instructions for shell execution, and supports both individual tasks and parallel evaluation runs.
Tradeoffs
Evaluations can fail when an agent-created tests directory changes Docker copy behavior. The documented remedy edits Harbor itself and may stop applying if Harbor changes. Installation also pulls Node through nvm and installs the latest pi package by default, which reduces environment reproducibility unless a version is supplied.
How to use it well
Use it when comparing pi and model combinations under Terminal-Bench, especially when you need structured usage and cost records. Start with one task and the oracle validation before scaling locally or through Daytona. It does not replace Harbor, fix Harbor automatically, or provide a general coding-agent interface.
Technical notes+
The Python package is configured in pyproject.toml for Python 3.11 or newer, uses Hatchling, and depends on Harbor. src/pi_terminal_bench/pi_agent.py defines PiAgent, builds Harbor ExecInput commands, invokes pi with --print --mode json, and parses message_end records from pi-output.jsonl into AgentContext. src/pi_terminal_bench/install-pi.sh.j2 installs Node 22 through nvm and installs either a requested pi version or latest. show-results.js parses result.json, calculates accuracy and standard error, and compares results with an embedded leaderboard. ERROR.md documents the Docker copy failure and proposed Harbor fixes.
Observed
- License
- MIT
- Primary language
- Python, with supporting JavaScript and shell scripts
- Python requirement
- Python 3.11 or newer
- Packaging
- Hatchling wheel containing the pi_terminal_bench package
- Install surface
- Development installation uses uv and an editable package with optional pytest and Ruff dependencies
- Interface
- Python agent class imported by Harbor; supporting shell runner and Node command-line results viewer
- Execution environments
- Local Docker and cloud execution through Daytona are documented
- Provider credentials
- The adapter forwards credentials for Anthropic, OpenAI, Google, Groq, xAI, and OpenRouter
Read from README.md, pyproject.toml, src/pi_terminal_bench/__init__.py, src/pi_terminal_bench/pi_agent.py, show-results.js, run.sh, ERROR.md, src/pi_terminal_bench/install-pi.sh.j2.
What it can do
Execute Terminal-Bench evaluation tests
Test scenarios and parameters → Evaluation results and metrics
Interface pi coding agent with Terminal-Bench
Pi coding agent requests → Formatted Terminal-Bench commands
Run terminal command evaluations
Terminal commands and expected outputs → Pass/fail results and performance data
Translate harbor agent protocols
Harbor agent messages → Terminal-Bench compatible formats
Monitor coding agent performance
Agent execution data → Performance benchmarks and scores
Generate benchmark reports
Evaluation session data → Structured benchmark reports
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.