Promptfoo
github.com/promptfoo/promptfoo- Category
- Developer Tools
- Rank
- No. 349Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- promptfoo
- GitHub
- 24.3k stars
- Latest release
- 0.122.0
- Added
- Jul 4, 2026
About
Promptfoo is an open-source tool for testing and evaluating LLM apps — declarative test cases, side-by-side model comparison, red-teaming, and CI/CD integration. Used by OpenAI and Anthropic.
What it does
Promptfoo runs repeatable checks across prompts and model providers, then records pass counts, failures, latency, token use, cost, and named metrics. It can generate adversarial cases, scan for vulnerabilities, and present results through a terminal or local web interface.
Why it's ranked here
This is a strong choice for engineering teams that need evaluation and security work in one system. The same package exposes command-line workflows, a Node library, an MCP command, CI automation, provider comparison, result sharing, and a local viewer. Its MIT license keeps adoption and modification straightforward.
What's good
The evaluation model captures more than a single quality score: schemas include assertion outcomes, errors, latency, token usage, cost, and named metrics. Prompt inputs can come from text, structured data, templates, JavaScript, Python, shell scripts, or executables. Red-team plugins, strategies, graders, and extraction tools are also exposed through the library.
Tradeoffs
The npm and npx routes require Node.js 22.22.0 or newer. Most model providers still require their own API keys, so local evaluation does not remove external service setup. The broad command surface, provider options, red-team configuration, local server, database migrations, and optional dependencies create meaningful operational complexity for small prompt experiments.
How to use it well
Use it when prompts or model choices need repeatable regression checks before release. Keep representative cases in source control, compare providers, inspect failures locally, and gate CI on assertions. Add red-team runs for security-sensitive changes. It does not supply model access itself, so retain provider credentials, budgets, and production monitoring elsewhere.
Technical notes+
package.json defines an ESM package with CommonJS compatibility, typed exports, CLI aliases, an MCP script, workspaces, and separate integration, smoke, browser, coverage, and package-artifact tests. src/main.ts registers evaluation, red-team, code-scan, model-scan, generation, validation, sharing, and viewer commands. src/index.ts exports evaluate, provider loaders, assertions, caching, guardrails, and a structured red-team library surface. src/server/server.ts builds an Express and Socket.IO local service with schema-validated routes. src/blobs/index.ts couples filesystem blob storage to database references and checks provenance before sharing. src/prompts/index.ts dispatches prompt loading by format, glob, function, or executable.
Observed
- License
- MIT
- Primary language
- TypeScript
- Install surfaces
- npm, npx, Homebrew, and pip
- Interfaces
- Command-line tool, Node library, MCP command, local web interface, and HTTP API
- Runtime requirement
- Node.js 22.22.0 or newer for npm and npx usage
- Package compatibility
- ES modules with CommonJS entry points and TypeScript declarations
- Local storage
- Filesystem-backed media and blob storage are the open-source defaults
- Test tooling
- Vitest suites cover unit, integration, smoke, coverage, and package-artifact workflows
Read from README.md, package.json, src/main.ts, src/index.ts, src/node/index.ts, src/util/index.ts, src/blobs/index.ts, src/types/index.ts, src/util/server.ts, src/server/index.ts, src/prompts/index.ts, src/redteam/index.ts, src/server/server.ts, src/storage/index.ts.
Tags
Tech Stack
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.