ObserverBench
kwisatzh.github.io- Category
- Developer Tools
- Rank
- No. 2032Tools index
Previous survey · No. 1911 ·
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- GitHub
- 1 stars
- Latest release
- v0.1.0
- Date
About
A benchmark that tests whether mechanistic interpretability methods (activation probes, sparse-autoencoder features, causal edits) which read an AI model's internal activity actually improve downstream safety decisions, not just predictions. It includes an interactive walkthrough plus a full workbench with results on GPT-2, Qwen2.5/3.5, and Gemma covering monitoring and internal-edit tasks.
What it does
ObserverBench scores a method that guesses something hidden about a model, such as whether a proposed action is unsafe or what an internal edit will do, by the decision that guess produces. Each task fixes the target, what the method may read, the allowed actions, the decision rule and the budget. You bring a score table plus a short card describing access and cost; the benchmark converts scores into allow, block or escalate choices and reports both ranking accuracy and the harm left over.
Why it's ranked here
It measures something most interpretability benchmarks skip, and its own results show why that matters. On a recorded Gemma-2-9B-it panel, a detailed prompted monitor beats the Gemma Scope sparse-autoencoder probe on AUROC (0.893 against 0.870) yet leaves more violations unaudited (10.80 against 9.07). A paper, a claim-to-result reproduction map and a DOI-archived release back the numbers. The ceiling is maturity: version 0.1.0, one author, and sealed public scoring is switched off, so the public rankings are built from the author's own runs.
What's good
The first run costs almost nothing: Git, Python 3 and make run two scored examples with no GPU, no model download and no API key. Public submissions are plain CSV and JSON that are checked as untrusted data, and contributor code is never executed. Every task declares its target, information boundary, actions, loss and operating conditions, so comparisons stay within one task version. The documentation states its limits plainly, including no robustness claim against adaptive attackers and practice scores kept out of the research rankings.
Tradeoffs
Submitting today buys a format check, not a score: sealed scoring is inactive and only one blinded safety pack accepts entries. The license is ambiguous, since the README says Apache 2.0 while the license file carries MIT text. The full install pulls PyTorch as a core dependency even for table-only tasks. In the evidence-integrity task the generator is public, so a submitter could regenerate the answers. There is no hosted runner: you run your own model and measurement hooks, and the notebooks reproduce named studies rather than serving as a general harness.
How to use it well
It suits interpretability and AI-control researchers asking whether a probe, attribution method or monitor actually improves a decision under a fixed budget. Run the two-example demo first, then the open practice tasks, which score locally against public answers. Compare results only within the same task version and access level, as the project itself insists. It does not evaluate your production model as a service, and it does not test resistance to attackers who adapt to the monitor. Treat any public submission as a file check until sealed scoring is turned on.
Technical notes+
pyproject.toml requires Python 3.10 or newer with core dependencies matplotlib, numpy, pandas and torch; extras add transformer-lens, exactly pinned transformers, scikit-learn and pytest. The observerbench console script maps to src/observerbench/cli.py, which exposes list, run, make-card and make-figures commands plus evaluate-effect-csv (a --no-verify-hashes flag skips SHA-256 checks), evaluate-safety-csv and evaluate-ai-control-csv (default --audit-budget 0.02). docs/API_CONTRACT.md defines a structurally typed v0 contract with a closed task registry and no plugin discovery. docs/PUBLIC_SUBMISSION_WORKFLOW.md keeps the private evaluator behind the SEALED_EVALUATOR_ENABLED variable, off at launch. scripts/reproduce_paper_fast.py rebuilds figures from checked summaries on CPU. LICENSE holds MIT text; README.md names Apache 2.0.
Observed
- License
- Conflicting: LICENSE holds MIT text, README.md states Apache License 2.0
- Language
- Python, requires 3.10 or newer
- Interfaces
- Command-line tool and Python library contract
- Install
- Editable pip install from a source checkout; demo needs only Git, Python 3 and make
- Hardware
- Demo and paper figure rebuild run on CPU without model downloads
- Submission format
- CSV and JSON data files; contributor code is not executed
- Models covered
- GPT-2-small, Qwen2.5-7B, Gemma-2-9B-it, Qwen3.5-9B
- Archive
- Release archived on Zenodo with a DOI
Read from README.md, pyproject.toml, LICENSE, src/observerbench/cli.py, docs/API_CONTRACT.md, docs/PUBLIC_SUBMISSION_WORKFLOW.md, docs/evidence_integrity.md, docs/RUNNERS.md, docs/SAFETY_TRACK.md, scripts/reproduce_paper_fast.py.
What it can do
Benchmark interpretability methods (activation probes, SAE features, causal edits) against downstream safety decisions
AI model internals/interpretability method outputs → Safety-decision performance results
Run monitoring tasks on models including GPT-2, Qwen2.5/3.5, and Gemma
Model activations → Monitoring task results
Run internal-edit tasks on models including GPT-2, Qwen2.5/3.5, and Gemma
Model activations → Internal-edit task results
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.