Crawl4AI
github.com/unclecode/crawl4ai- Category
- AI Tools
- Rank
- No. 20Tools index
- Type
- APP
- Builder
- unclecode
- GitHub
- 83.5k stars
- Latest release
- v0.9.3
- Date
About
An open-source, LLM-friendly web crawler and scraper that turns websites into clean, structured data for AI pipelines.
What it does
Crawl4AI drives real or remote browsers, renders dynamic pages, and converts their contents into Markdown, metadata, media records, links, tables, or schema-shaped JSON. It supports focused single-page work, concurrent batches, deep exploration, saved browser identities, caching, and selector-based or model-assisted extraction.
Why it's ranked here
Its breadth is unusually practical: the same package handles browser sessions, dynamic scrolling, link discovery, content filtering, structured extraction, batch crawling, and self-hosted serving. The strongest case is control without a required account or API key. The security history and substantial dependency surface still demand careful deployment.
What's good
You can choose cheap deterministic extraction with CSS or XPath, then reserve language models for irregular content. Browser profiles preserve cookies and authentication across multi-step work. Hooks, proxies, headers, scripts, remote browser connections, screenshots, caching, crash recovery, and URL-specific configurations cover many production scraping problems without forcing separate tools.
Tradeoffs
Installation includes a large Python dependency set plus browser setup, so this is not a lightweight HTTP scraper. The Docker API previously exposed critical remote execution and local file access flaws; newer releases disable hooks and block file URLs there by default. The legacy packaging path also removes an existing local crawl cache during setup.
How to use it well
Pick it for Python pipelines that need rendered pages, repeatable extraction, authenticated sessions, or controlled multi-page discovery. Start with selectors and filtered Markdown, add model extraction only where structure varies, and isolate the Docker API behind authentication. It does not yet replace a generally available managed crawling service because the cloud API remains closed beta.
Technical notes+
pyproject.toml defines a setuptools package requiring Python 3.10+, with Playwright, Patchright, HTML parsers, BM25 tooling, Pydantic, HTTP clients, and optional PDF, Torch, transformer, cosine, and Selenium extras. Its console entry points expose crwl, setup, diagnostics, migrations, and model downloads. setup.py, retained for compatibility, creates .crawl4ai storage under the configured base directory or home directory and deletes the existing cache tree before recreating content folders. docs/codebase/browser.md describes context reuse through configuration signatures, session TTL cleanup, CDP connections, persistent profiles, and Chromium process monitoring. docs/RELEASE_NOTES_v0.8.0.md records that Docker hooks became disabled by default and API handling of file:// URLs was blocked after remote execution and local file inclusion vulnerabilities.
Observed
- License
- Apache-2.0
- Primary language
- Python
- Python support
- Python 3.10 or newer; classifiers list 3.10 through 3.13
- Install surface
- PyPI package installed with pip, followed by browser setup and diagnostic commands
- Interfaces
- Asynchronous Python library, command-line interface, and Docker-hosted FastAPI server
- Browser support
- Chromium, Firefox, WebKit, and remote Chrome DevTools Protocol connections
Read from README.md, setup.py, pyproject.toml, requirements.txt, docs/RELEASE_NOTES_v0.8.0.md, docs/md_v2/index.md, docs/codebase/cli.md, docs/codebase/browser.md, docs/blog/release-v0.7.0.md, docs/blog/release-v0.7.1.md, docs/blog/release-v0.7.3.md, docs/blog/release-v0.7.4.md, docs/blog/release-v0.7.5.md, docs/blog/release-v0.7.6.md.
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.