Market Data Warehouse
github.com/joemccann/market-data-warehouse- Category
- Finance
- Rank
- No. 1505Tools index
Previous survey · No. 1447 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- @joemccann
- GitHub
- 182 stars
- Date
About
A local-first financial data warehouse built for quantitative analysts and traders to store, analyze, and backtest market data across equities, futures, and volatility indices. Uses Parquet for storage, DuckDB for analytics, and supports production workflows with ClickHouse.
What it does
It collects daily price and volume bars, keeps one canonical snapshot per symbol, and rebuilds an analytical database when needed. Ingestion can fill historical gaps, validate records, preserve existing history, and replace snapshots atomically. Separate schemas handle equities, volatility products, and futures.
Why it's ranked here
The design makes a strong research foundation because canonical files remain independent of the analytical database, and failed updates should not expose partial snapshots. The verdict is mixed for heavier operations: current merges rewrite each symbol’s full history, while the repository’s own blueprint identifies missing leases, staged manifests, and stronger run coordination.
What's good
Atomic publication is concrete, not aspirational: temporary Parquet output is read back, checked, then replaced in one filesystem operation. Stable symbol identifiers survive rebuilds. Gap-only updates reduce unnecessary vendor requests, while Nasdaq and Stooq form an ordered recovery chain when Interactive Brokers misses a daily equity bar.
Tradeoffs
Setup carries substantial operational weight. Interactive Brokers ingestion needs an account and a running gateway, with two-factor authentication still part of the workflow. The documented environment targets modern macOS tooling. Full-history snapshot rewrites make small updates increasingly expensive, and cursor state does not prevent overlapping backfills or track window-level ownership.
How to use it well
Use it when a Python-oriented quant workflow needs reproducible daily datasets that remain queryable without a permanently running database service. Keep Parquet authoritative, rebuild DuckDB for local analysis, and add ClickHouse only for concurrent or larger queries. Treat it as data infrastructure, not a finished strategy, attribution, or visualization framework.
Technical notes+
clients/bronze_client.py defines PyArrow schemas for equity, volatility, and futures snapshots, merges rows by trade date, writes Snappy Parquet to a temporary sibling, validates it, and publishes with os.replace(). clients/db_client.py creates DuckDB tables and unique indexes, but its daily inserts iterate row by row and skip constraint conflicts. clients/ib_client.py wraps ib_insync, retries successive client IDs after error 326, and includes portfolio, market-data, and order operations. clients/daily_bar_fallback.py queries Nasdaq stocks, then ETFs, then Stooq, with throttling and retry handling. pyproject.toml requires Python 3.12 and configures pytest plus 100 percent coverage enforcement for clients and scripts, excluding clients/ib_client.py. package.json is a private Node 22 alert package whose sole runtime dependency is Nodemailer. docs/observability_defensive_blueprint.md explicitly records remaining full-snapshot rewrite, cursor ownership, leasing, and staged-manifest gaps.
Observed
- License
- MIT
- Primary language
- Python, with a small private Node.js alerts component
- Installation surface
- Shell bootstrap script, Python 3.12 or newer, and npm for alerting dependencies
- Interfaces
- Command-line ingestion and maintenance scripts plus importable Python clients
- Platform support
- macOS is listed as required, with Apple Silicon recommended
- Canonical storage
- Per-symbol Parquet snapshots compressed with Snappy
- Analytics engines
- DuckDB locally, with optional ClickHouse for aggregation and concurrency
- Data providers
- Interactive Brokers and CBOE for ingestion, with Nasdaq and Stooq fallback handling for daily U.S. equity bars
Read from README.md, package.json, pyproject.toml, docs/ib_tws_api.md, docs/backtesting.md, docs/observability_defensive_blueprint.md, clients/__init__.py, clients/db_client.py, clients/ib_client.py, clients/uw_client.py, clients/symbol_ids.py, clients/bronze_client.py, clients/daily_bar_fallback.py.
What it can do
Store historical OHLCV market data
Historical price data (Open, High, Low, Close, Volume) for equities, futures, and volatility indices → Organized data warehouse with Parquet file storage
Analyze market data with SQL queries
SQL queries and stored market data → Query results and analytics insights using DuckDB
Backtest trading strategies
Trading strategy parameters and historical market data → Backtesting results and performance metrics
Convert daily bar data to production analytics format
Daily OHLCV bar data → Production-ready analytics data structures
Export data to ClickHouse for production workflows
Processed market data from local warehouse → Data transferred to ClickHouse production environment
Process multi-asset class financial data
Raw market data from equities, futures, and volatility indices → Normalized and structured financial datasets
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.