Eliza DQ
github.com/se7enquick/eliza-dq- Category
- Developer Tools
- Rank
- No. 1910Tools index
Previous survey · No. 1811 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Data Processing
- Interfaces
- CLI · SDK
- GitHub
- 3 stars
- Latest release
- v0.2.1
- Date
About
Eliza DQ is an open-source Python data quality library that validates data across BigQuery, Athena, Snowflake, Postgres, and other warehouses as well as Polars/pandas dataframes, parquet, and CSV files through a single API and YAML config. Its README publishes benchmarks claiming it beats Soda Core on SQL pushdown speed and cost, and beats Pandera and Great Expectations on large-scale streaming validation (259M rows in under 2 seconds).
What it does
A rules engine for catching bad rows. You list checks per column, such as no blanks, no duplicates, values inside a range, or records no older than a day, and it counts the failures. Against a database it compiles the simple checks into one counting query, runs the heavier ones beside it on up to eight threads, then fetches a handful of offending rows for each failed check. Against local files or in-memory tables it builds the same counts as deferred queries that stream through the data in chunks. A command-line runner returns an exit code, so a pipeline can stop on bad input.
Why it's ranked here
The mechanism behind its headline claim is real and small enough to read: sample queries fetch only a key column plus the failing column, and they run in parallel, which is where the published speed and cost gaps over Soda Core come from. The core install pulls two libraries, it is MIT licensed, and continuous integration runs unit tests on four Python versions plus live Postgres and MySQL services. Against that, the package labels itself alpha, and every benchmark figure is the author's own, run on the author's tables.
What's good
Failure output is useful rather than a bare pass or fail: each check reports a count, a rate and sample rows, and the result exports as JSON with an exit code for CI. The profiler scans a file and drafts a starter rule set: not-null and unique where the data already holds, a numeric range padded by ten percent, and allowed values for text columns with twenty or fewer distinct entries. Warehouse drivers are optional extras, so a Snowflake user never installs the BigQuery client. Values in allowed-list checks are quoted and escaped before they reach the SQL.
Tradeoffs
Table names, column names, row filters, range bounds and custom expressions from the config are pasted straight into SQL text without escaping, so the config file must be treated as trusted code. The duplicate check counts different things per engine: extra duplicate rows on the dataframe path, distinct repeated values on the warehouse path. The local mode for production databases runs a full select on the filtered table and holds the whole result in memory, so the filter carries all the load. Sample collection on the dataframe path swallows errors silently. CI has no jobs for BigQuery, Snowflake, Athena or the other cloud warehouses.
How to use it well
Fits a data engineer who wants a nightly gate on warehouse tables billed per byte scanned, or a quick validation step on parquet files inside a CI runner or scheduled job. On wide tables, name a narrow key column for samples, since that is what keeps sample queries cheap. Keep configs in version control and review them like code, given how they reach the database. Always set a filter in local mode. Treat the profiler's output as a draft: its ranges come from today's data, not from business rules.
Technical notes+
check_sql in src/eliza/sql_runner.py batches inline checks into one aggregation SELECT and dispatches it alongside unique, freshness, row_count and reference queries through a ThreadPoolExecutor capped at 8 workers; sample columns fall back from sample_columns to sample_id to every checked column. src/eliza/checks_sql.py applies _sql_quote only to in_set and not_missing values; table, column, filter and custom_sql text is f-string interpolated. unique is count minus n_unique in src/eliza/checks_polars.py but a count of duplicated values in SQL. src/eliza/runner.py swallows Polars sampling errors with a bare except. pyproject.toml declares 0.2.1 and Alpha while src/eliza/__init__.py says 0.1.0, and its project URLs name a different repository slug.
Observed
- License
- MIT
- Language
- Python, 3.10 or newer
- Core dependencies
- polars and pyyaml only
- Interfaces
- Python library and a CLI with check, init and learn commands
- Warehouse connectors
- BigQuery, Athena, Snowflake, Postgres, ClickHouse, MySQL, Databricks, Redshift as optional pip extras
- Declared maturity
- Development Status 3, Alpha, in pyproject.toml
- CI coverage
- Unit tests on Python 3.10 to 3.13; integration jobs for Postgres, MySQL, DuckDB and Polars only
- Packaging
- PyPI package eliza-dq
Read from README.md, pyproject.toml, LICENSE, src/eliza/__init__.py, src/eliza/cli.py, src/eliza/runner.py, src/eliza/checks_polars.py, src/eliza/checks_sql.py, src/eliza/sql_runner.py, src/eliza/learn.py, src/eliza/connectors/postgres.py, examples/quickstart.py, examples/eliza_checks/orders.yaml, .github/workflows/ci.yml.
What it can do
Validate data quality across data warehouses (BigQuery, Athena, Snowflake, Postgres)
Warehouse tables via YAML/inline config → Data quality check results
Validate data in dataframes and files (Polars, pandas, parquet, CSV)
Dataframe or file (parquet/CSV) → Data quality check results
Run built-in data quality checks
Dataset and check configuration → Pass/fail check results
Send data quality alerts
Check results → Slack message, PDF report, or webhook notification
Integrate data quality checks into orchestration pipelines
Pipeline definitions (Airflow, Dagster, GitHub Actions) → Automated data quality validation runs
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.