- Category
- Developer Tools
- Rank
- No. 1288Tools index
Previous survey · No. 1270 ·
- Type
- TOOL
- Use case
- Software Testing & Quality
- Interfaces
- CLI
- GitHub
- 110 stars
- Latest release
- v0.3.1
- Date
About
Assay is a QA CLI tool that finds bugs in web pages without requiring written tests or an LLM. It drives the page itself in a real browser via Playwright and Chromium, clicking every control, and flags places where the page contradicts itself, which the author treats as a reliable signal of a bug. It ships with reproducible benchmark suites in the repository comparing its findings against hand-checked and deliberately planted bugs.
At a glance
Assay opens a built web page in Chromium, tries controls it finds and reports failures. It is worth running on a local prototype when you want a quick check for interactions that appear broken before writing workflow-specific tests.
Good fit
Developers and coding agents checking one locally built page with buttons, inputs or draggable controls.
Consider something else when
Skip it if you need to log in, traverse several pages or prove the application returns the correct business result.
Start here
Point Assay at a local fixture with a working button and a deliberately inert one. Read the report to see which behavior it flags, then add an explicit test for the expected result.
Cost and requirements
MIT-licensed. Requires Python 3.10 or later and Chromium, which the first run can download. It needs built page output and no model API key.
Limits to know
The tool infers checks from page controls and can miss an interaction that responds but returns the wrong result. The page keeps network access during the run. Use a fixture disconnected from production services.
Sources checked 2026-11-05. Source-based assessment.
Read the source review
What it does
Point Assay at a folder holding a finished web page and it opens that page in a headless Chromium, lists every control the browser renders, including drag handles and clickable divs, then plans and runs checks against each one in a fresh tab. Because it has no idea what the page is meant to do, it does not judge intent. It looks for internal contradictions instead: a thrown error, an Undo that only works on the second press, a counter showing one item in an empty list, a NaN where a number belongs. Output is a terminal report, a single summary sentence, JSON, or an HTML report with screenshots.
Why it's ranked here
It fills a gap the usual tools leave open: checking a page nobody has written tests for, with no reference screenshot and no model call, so the same page gives the same result every run. On 225 AI-written pages a person had tried by hand, it caught the real bug on 15 of the 20 broken ones and flagged none of the 205 working ones. On deliberately planted bugs it found 12 of 50, and the README calls that score in-sample. A low false-alarm rate with modest recall is the right shape for something that runs after every agent turn.
What's good
The agent-facing details are thought through. The one-line summary is assembled by the tool, not by the agent, and any text pulled from the page is flattened to a single printable line so a hostile page cannot slip in a fake "nothing flagged". Checks that could not run are counted and reported rather than quietly treated as passes. It refuses to run npm install or a build on code nobody has read, and instead spots an unbuilt entry page and prints the build command or points at an existing dist folder. Several failures caused by one bug are folded into a single finding.
Tradeoffs
It cannot catch a page that works but does the wrong thing: a total that adds incorrectly usually passes unless you supply acceptance criteria from Python. It checks one page at a time from a local folder, does not crawl or log in, and leaves the page's internet access on, so a live API's responses shape the result. Built output only, so framework projects need a build first. A finished game with no restart can read as a dead page. Recall on planted bugs is roughly a quarter, so a clean result is weak evidence of a correct page. The package declares itself beta.
How to use it well
Its best fit is as a guardrail on coding agents that generate small, self-contained pages: install the plugin in Claude Code or add the skill to another harness, and every turn that touches a page ends with a one-line verdict the agent repeats verbatim. In CI or scripts, rely on the exit code: zero clean, one for a failed check, two when the page could not be checked. Write the HTML report when you need screenshots per check. Keep your real end-to-end tests for login flows, multi-page journeys and business rules, which it does not reach.
Technical notes+
pyproject.toml publishes assay-ui (Python >=3.10, MIT, classifier Development Status 4 - Beta) with a single runtime dependency, playwright>=1.40, pytest as the only dev extra, and a console script assay = assay.cli:main. In src/assay/cli.py, ensure_browser() probes for Chromium in a subprocess (to avoid a second in-process Playwright leaving tasks pending) and runs python -m playwright install chromium on a miss. unbuilt() regex-scans the first 8000 chars of the entry page for a script src under /src or /app ending in jsx, tsx, ts, vue or svelte, then suggests dist, build, out, .output/public or public if one holds a built index; _build_it() explains the refusal to run installs. _flat() collapses page-derived labels and errors to one printable line before one_line() emits them, and one_line() reports how many planned cases did not reach PASSED or FAILED. main() returns 0, 1 or 2, with --if-page returning 0 on uncheckable input for hooks. as_json() includes console errors and failed requests but never fails a case on them. src/assay/__init__.py exports check, look, plan and QA as the Python API and reads the version from package metadata. bench/machine.py records CPU, OS, Python and Chromium/Playwright versions next to every published timing. .claude-plugin/marketplace.json registers the Claude Code plugin from ./plugins/assay.
Observed
- License
- MIT
- Language
- Python, requires 3.10 or newer
- Install
- pip install assay-ui; uv tool and pipx also documented
- Runtime dependencies
- Playwright only; Chromium is downloaded on first run
- Interfaces
- CLI, Python API, agent skill, Claude Code plugin with an end-of-turn hook
- Output formats
- Terminal report, one-line summary, JSON, HTML report with screenshots
- Declared maturity
- Beta (package classifier in pyproject.toml)
- Scope
- One locally served page per run; no crawling, no login, no build step
- Benchmark tooling
- Benchmark suites live in the repository; bench/machine.py records the machine for each timing
Read from README.md, pyproject.toml, src/assay/cli.py, src/assay/__init__.py, bench/machine.py, .claude-plugin/marketplace.json.
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.