Prompt Replay
https://github.com/yeachan-heo/prompt-replay- Category
- Developer Tools
- Rank
- No. 1791Tools index
Previous survey · No. 1732 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- Yeachan-Heo
- GitHub
- 6 stars
- Date
About
AI prompt regression testing framework — record, replay, and compare LLM responses.
What it does
Prompt Replay currently provides a command-line recorder for named prompt cases. It accepts prompt text from an option or standard input, stores an optional expected answer, and saves each case in a local SQLite database.
Why it's ranked here
The implemented core is much narrower than the stated regression-testing ambition. Recording is concrete and usable, but replay and comparison exist only as data shapes and database capacity, not registered command-line workflows.
What's good
The recorder supports both direct text and piped input, making it easy to fit into shell workflows. Local SQLite storage avoids a hosted service, while UUIDs and timestamps give saved cases stable identities.
Tradeoffs
Only the record command is registered. There is no implemented replay, model invocation, response comparison, run capture, reporting, or test automation in the supplied source. The package is also marked private, which limits its current distribution story.
How to use it well
It best suits TypeScript developers prototyping a local prompt-case inventory from scripts or terminal sessions. Use it to capture prompts and expected text before building a fuller harness. It does not yet replace an LLM runner, evaluator, regression suite, or continuous-integration gate.
Technical notes+
src/cli.ts creates the Commander program and registers only registerRecordCommand. src/commands/record.ts initializes storage, reads --prompt or stdin, generates a UUID, and inserts into tests. src/db/schema.ts creates tests and runs tables plus indexes in ~/.prompt-replay/prompts.db, although no supplied command writes runs. src/types.ts defines PromptTest, TestRun, and ComparisonResult, including token usage, latency, diffs, and cost comparison fields. package.json marks the package private, exposes a CLI bin entry, builds src/cli.ts as ESM with declarations, and provides development and type-check scripts. tsconfig.json enables strict TypeScript checking with no emit.
Observed
- Primary language
- TypeScript
- Interface
- Command-line interface built with Commander
- Implemented command surface
- One registered command records named prompt cases
- Storage
- Local SQLite database under the user home directory
- Package surface
- Private ESM package with a prompt-replay CLI bin entry
- Toolchain
- Bun-oriented setup with tsup, tsx, and strict TypeScript checking
Read from README.md, package.json, src/cli.ts, src/types.ts, src/db/schema.ts, src/commands/record.ts, bun.lock, CLAUDE.md, tsconfig.json, .cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc.
What it can do
Record LLM prompt-response pairs
AI prompts and corresponding LLM responses → Recorded test cases with prompt-response data
Replay recorded prompts against LLM
Recorded test cases and LLM endpoint → New LLM responses for comparison
Compare LLM responses across test runs
Original and new LLM response pairs → Difference analysis and regression test results
Execute regression test suites
Test suite configuration and LLM model → Pass/fail results and performance metrics
Track LLM response consistency over time
Historical test run data → Consistency reports and trend analysis
Validate LLM model changes
Before and after model versions with test prompts → Model validation report with response differences
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.