Scenario
github.com/langwatch/scenario- Category
- Developer Tools
- Rank
- No. 434Tools index
- Pricing
- Open Source
- Platform
- cli
- Type
- TOOL
- Builder
- langwatch
- GitHub
- 951 stars
- Latest release
- python/v1.3.0
- Added
- Jul 4, 2026
About
Scenario is an agent testing framework that uses LLM-powered user simulators to run end-to-end simulations of AI agents across diverse scenarios and edge cases. It enables developers to validate tool calling, multi-turn conversations, and agent behavior without requiring pre-built datasets. Works with LangGraph, CrewAI, Pydantic AI, and any other agent framework.
What it does
Scenario turns a plain-language situation into an executable agent test. You connect your agent through a small adapter, add a simulated user and optional judge, then choose between automatic conversation or a scripted sequence. Scripts can inject messages, check tool use, call custom evaluators, and stop with an explicit verdict.
Why it's ranked here
The design covers both exploratory conversations and precise regression checks without forcing teams into one agent stack. Its strongest argument is control: tests can start scripted, continue automatically, and inspect intermediate behavior. Python and TypeScript examples fit established test runners. The broader voice subsystem adds ambition, but also meaningful installation weight and configuration complexity.
What's good
A single adapter method is a small integration boundary. Tests can mix generated dialogue with fixed messages, custom assertions, external evaluation tools, and judge criteria. The state can verify tool activity during the exchange, not merely inspect a final answer. Per-call configuration isolates concurrent executions, while environment variables remain a fallback for simpler command-line use.
Tradeoffs
Automated scenarios require model credentials, and judge results depend on criteria supplied by the test author. Voice support makes codecs a core dependency: Python and JavaScript bundle an ffmpeg binary even for text-only consumers, adding roughly 30 to 80 MB at installation. The repository also documents unresolved Python parity work around per-run voice provider isolation.
How to use it well
Use Scenario for teams shipping conversational or tool-using agents that need repeatable behavior checks in existing Python or TypeScript test suites. Keep critical steps scripted, use generated turns for variation, and attach direct assertions where outcomes are machine-checkable. It tests an agent you provide. It does not build, host, or supply that agent.
Technical notes+
README.md presents Python, TypeScript, and Go support, with pytest and Vitest quick starts. package.json exposes CommonJS, ESM, declaration files, and Vitest reporter, setup, global setup, and config subpaths, although the repository-root package is marked private. javascript/src/index.ts exports runner, script, domain, execution, agent, tracing, red-team reporting, and voice factory surfaces. python/scenario/cli.py installs a scenario console entry point whose current command launches a Streamlit red-team report dashboard. docs/adr/001-scenario-concurrency-model.md specifies per-call EventBus and LangWatch configuration for concurrent isolation. docs/adr/004-ffmpeg-bundling.md records hard bundled ffmpeg dependencies for both SDKs, native WAV handling, and subprocess transcoding for MP3, OGG, and FLAC.
Observed
- Languages
- Available in Python, TypeScript, and Go.
- Package installs
- Python installs as langwatch-scenario; JavaScript installs as @langwatch/scenario.
- Library formats
- The JavaScript package exposes CommonJS, ESM, and TypeScript declarations.
- Test runner integrations
- Quick starts use pytest and Vitest, with dedicated Vitest integration exports.
- CLI
- The Python package provides a scenario console script for opening red-team reports.
- Configuration model
- Concurrent runs use per-call configuration and separate event buses, with environment variables as fallback.
- Voice codecs
- Python and JavaScript bundle ffmpeg dependencies; WAV is native while MP3, OGG, and FLAC use transcoding.
Read from README.md, Makefile, package.json, docs/adr/004-ffmpeg-bundling.md, docs/adr/002-voice-provider-state.md, docs/adr/003-voice-internal-design.md, docs/adr/001-scenario-concurrency-model.md, docs/vocs.config.tsx, docs/eslint.config.ts, javascript/eslint.config.mjs, javascript/demo-sliding-deadline.ts, docs/docs/layout.tsx, python/scenario/cli.py, docs/docs/constants.ts, javascript/src/index.ts.
What it can do
Simulate realistic user interactions with AI agents
Agent definition and test scenario description → Multi-turn conversation simulation between LLM-powered user and agent
Validate agent tool calling behavior
Agent with tool integrations and test scenario → Pass/fail results on whether correct tools were called with correct parameters
Test agents across edge cases without pre-built datasets
Scenario descriptions and agent under test → Automated edge case conversations and behavioral test results
Evaluate multi-turn conversation handling
Agent and multi-step dialogue scenario → Assessment of agent responses across multiple conversation turns
Run custom evaluations on agent behavior
Agent responses and user-defined evaluation criteria → Scored or pass/fail evaluation results based on custom logic
Integrate agent testing into existing frameworks
LangGraph, CrewAI, Pydantic AI, or custom agent via a single call() method → Test execution and results without framework-specific boilerplate
Run end-to-end agent simulations programmatically
Agent callable and scenario configuration in Python, TypeScript, or Go → Simulation transcript and structured test outcome
Intel on Scenario
Tags
Tech Stack
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.