- Category
- Developer Tools
- Rank
- No. 399Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- explodinggradients
- GitHub
- 15.3k stars
- Latest release
- v0.4.3
- Added
- Jul 4, 2026
About
Ragas is an open-source evaluation framework specialized for RAG pipelines, scoring retrieval and generation with metrics like faithfulness, answer relevancy, and context precision.
What it does
Ragas turns LLM behavior into repeatable experiments. You can define custom criteria, generate test data, run evaluations asynchronously, compare results with baselines, and store experiment data through memory, CSV, JSONL, or Google Drive backends.
Why it's ranked here
The project covers more of the evaluation workflow than a metric collection alone. Test generation, custom categorical and numeric criteria, baseline comparisons, caching, experiment versioning, and framework integrations make it useful for teams building an evaluation practice, not just running occasional checks.
What's good
The metric surface spans factual checks, string matching, rubrics, multimodal outputs, tool calls, agent goals, ranking, and custom criteria. It supports single-turn and multi-turn samples. Provider wrappers, prompt models, local storage backends, caching, and tracing integrations give engineers useful extension points.
Tradeoffs
The default installation pulls in OpenAI and several LangChain packages, even when a narrower setup might suffice. Many advanced capabilities require optional dependency groups. Several legacy metric, embedding, and model wrapper imports are deprecated, so existing integrations may need migration. Minimal anonymized analytics are enabled unless users opt out.
How to use it well
Use Ragas when a Python team wants evaluation datasets, reusable criteria, experiment comparisons, and feedback loops around an LLM application. Start with the RAG project template, then add domain-specific criteria and production-derived cases. Pair it with an observability tool when tracing and operational monitoring are also required.
Technical notes+
pyproject.toml defines a setuptools package under src, Python >=3.9, a ragas Typer CLI entry point, backend entry points, and optional groups for tracing, Google Drive, Haystack, OCI, AG-UI, DSPy, and broader metric dependencies. src/ragas/__init__.py exports synchronous and asynchronous evaluation, datasets, experiments, caching, run configuration, and tokenizers. src/ragas/metrics/__init__.py exposes metric primitives while routing many older imports through deprecation warnings. src/ragas/cli.py aggregates numeric values by mean, categorical values by frequency, and renders baseline comparisons with Rich. Makefile defines Ruff, Pyright, unit, end-to-end, notebook, benchmark, and documentation workflows. README.md documents opt-out analytics through RAGAS_DO_NOT_TRACK.
Observed
- License
- Apache-2.0
- Primary language
- Python
- Python support
- Python 3.9 or newer
- Installation
- PyPI package via pip, with direct Git repository installation also documented
- Interfaces
- Python library and Typer-based command-line interface
- Storage backends
- In-memory, local CSV, local JSONL, and optional Google Drive backends
- Build system
- setuptools with setuptools-scm and a src-layout package
- Quality structure
- Repository automation includes formatting, linting, basic type checking, unit tests, end-to-end tests, notebook tests, and benchmarks
Read from README.md, Makefile, pyproject.toml, src/ragas/cli.py, src/ragas/__init__.py, src/ragas/llms/__init__.py, src/ragas/prompt/__init__.py, src/ragas/metrics/__init__.py, src/ragas/testset/__init__.py, src/ragas/backends/__init__.py, src/ragas/embeddings/__init__.py, src/ragas/optimizers/__init__.py, src/ragas/integrations/__init__.py, src/ragas/llms/adapters/__init__.py, src/ragas/prompt/metrics/__init__.py.
Tags
Tech Stack
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.