MCP Crawl4AI RAG
github.com/coleam00/mcp-crawl4ai-rag- Category
- Developer Tools
- Rank
- No. 504Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- coleam00
- GitHub
- 2.3k stars
- Date
About
Cole Medin's MCP server for web crawling and RAG — gives AI coding assistants the ability to fetch, parse, and index live documentation.
What it does
It builds a searchable knowledge base from websites. The server detects sitemaps, text files, and ordinary pages, crawls them appropriately, splits content into chunks, creates embeddings, and stores the results in Supabase. Agents can then search by meaning, keywords, source domain, or dedicated code examples.
Why it's ranked here
The appeal is breadth within one MCP service: crawling, chunking, retrieval, reranking, code-example search, and optional repository graphs. Those capabilities come with substantial infrastructure and maturity caveats. The author calls the repository a testbed, and the more ambitious graph features are not fully compatible with its recommended Docker setup.
What's good
URL-aware crawling removes manual routing between single pages, sitemaps, text documents, and recursively discovered links. Retrieval can combine semantic and keyword matches, filter by source, rerank results, and keep extracted code examples in a separate search path. Batch embedding, parallel crawling, retries, and duplicate replacement address practical indexing work.
Tradeoffs
The baseline requires Supabase and an OpenAI API key, while graph analysis adds Neo4j. Embeddings use one hardcoded OpenAI model, so the promised provider flexibility is not present yet. Contextual indexing and code extraction add LLM calls, latency, and storage. Failed embeddings can become zero vectors, which preserves ingestion but may quietly weaken retrieval.
How to use it well
Use it when an MCP-capable coding agent needs a maintained search layer over documentation and your team accepts operating Supabase, API credentials, and optional Neo4j. Start with ordinary crawling and semantic search, then enable hybrid search or reranking when retrieval demands justify the overhead. It does not cover fully local, provider-neutral RAG today.
Technical notes+
pyproject.toml defines Python 3.12 or newer and pins Crawl4AI, MCP, Supabase, and OpenAI dependencies while allowing broader sentence-transformers and Neo4j versions. src/crawl4ai_mcp.py creates a FastMCP server with an async crawler lifecycle, optional CrossEncoder reranking, SSE-oriented environment configuration, and conditional Neo4j initialization. src/utils.py hardcodes text-embedding-3-small, retries batch embeddings three times, falls back to individual calls, then inserts 1,536-element zero vectors for individual failures. Its contextual processing appends futures in completion order without restoring their original indices before pairing content with URLs and metadata, so concurrent completion can misassociate chunks. knowledge_graphs/parse_repo_into_neo4j.py parses Python ASTs directly into repository, file, class, method, function, attribute, and import structures without LLM processing. knowledge_graphs/ai_script_analyzer.py analyzes Python scripts only, while knowledge_graphs/hallucination_reporter.py produces structured validation reports.
Observed
- Primary language
- Python
- Runtime
- Python 3.12 or newer
- Install surfaces
- Docker image build or editable installation with uv
- Interface
- Model Context Protocol server, configured for SSE transport in the documented environment settings
- Required services
- Supabase with pgvector and an OpenAI API key
- Optional service
- Neo4j enables repository graphs and Python script hallucination checks
- Core packaging
- Python project metadata declares Crawl4AI, MCP, Supabase, OpenAI, sentence-transformers, and Neo4j dependencies
Read from README.md, pyproject.toml, src/utils.py, src/crawl4ai_mcp.py, knowledge_graphs/test_script.py, knowledge_graphs/ai_script_analyzer.py, knowledge_graphs/parse_repo_into_neo4j.py, knowledge_graphs/query_knowledge_graph.py, knowledge_graphs/hallucination_reporter.py.
What it can do
Crawl web pages for documentation
Website URLs or documentation links → Parsed web page content
Parse HTML documentation into structured data
Raw HTML content from web pages → Structured text and metadata
Index documentation content for retrieval
Parsed documentation text → Searchable content index
Fetch live documentation updates
Documentation source URLs → Current version of documentation content
Provide RAG capabilities to AI coding assistants
User queries and indexed documentation → Contextually relevant documentation snippets
Extract and process API documentation
API documentation web pages → Structured API reference data
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.