- Category
- Developer Tools
- Rank
- No. 1362Tools index
Previous survey · No. 1352 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- amantus-ai
- GitHub
- 346 stars
- Date
About
Transforms developer documentation into clean LLM-ready Markdown, complete with llms.txt support.
What it does
LLM Codes crawls documentation websites whose useful content appears only after JavaScript runs. It renders pages through hosted Firecrawl or self-hosted Chromium, follows matching links, removes navigation and repeated material, and combines the results into a downloadable document. Users control crawl depth, page limits, filtering, deduplication, and code-only extraction.
Why it's ranked here
The tool addresses a specific, common failure mode for coding agents with a practical browser-based pipeline. Its bounded parallel crawling, content checks, retries, caching, progress stream, and configurable cleanup show more depth than a basic page converter. The restricted URL matcher and provider requirements keep it from being a general web research system.
What's good
Users can choose hosted rendering or operate Chromium themselves. Crawls enforce depth and page limits, process batches concurrently, and report live progress. Cleanup can remove links, availability notices, boilerplate, empty sections, formatting artifacts, and duplicate passages. Code-only mode skips pages without fenced examples, which helps build compact reference material.
Tradeoffs
Input validation accepts recognized documentation domains, common documentation paths, programming-language sites, GitHub Pages, and explicit exceptions. It does not accept arbitrary websites. The default route requires a Firecrawl key, while self-hosting requires Chromium. Deep crawl mode belongs to the Firecrawl path. Distributed caching and durable retry behavior depend on Redis.
How to use it well
Use it when a coding agent cannot read dynamic API references or when you need a bounded, filtered snapshot across several documentation pages. Start with shallow crawls, then raise depth and page limits deliberately. Choose code-only output for example mining. Use another tool for arbitrary-site research, indexing, semantic search, or document retrieval.
Technical notes+
The Next.js client in src/app/page.tsx supports individual scraping and crawl jobs, while src/hooks/useCrawl.ts consumes server-sent progress events and applies documentation filters. src/lib/firecrawl.ts calls Firecrawl through the pooled HTTP/2 client in src/lib/http2-client.ts, validates returned content, and maps provider failures to user-facing errors. src/constants.ts defines URL patterns, crawl bounds, concurrency, timeouts, retries, and cache periods. src/lib/retry-queue.ts and src/lib/circuit-breaker.ts use Upstash Redis, with reduced behavior when Redis is absent. src/utils/file-utils.ts filters failed pages and assembles the browser download. package.json provides Vitest, coverage, lint, type-check, build, and combined verification scripts.
Observed
- License
- MIT
- Primary language
- TypeScript
- Install surface
- Source application installed with pnpm and run on Node.js 24 or newer
- Interfaces
- Browser interface plus HTTP scrape and crawl endpoints
- Rendering providers
- Firecrawl by default, with opt-in self-hosted Playwright and Chromium
- Deployment support
- Vercel deployment or self-hosted Node and Docker deployments
- Testing
- Vitest test, UI, API, coverage, and combined verification scripts are configured
Read from README.md, package.json, src/constants.ts, src/app/icon.tsx, src/app/page.tsx, src/lib/errors.ts, src/app/layout.tsx, src/lib/firecrawl.ts, src/hooks/useCrawl.ts, src/lib/retry-queue.ts, src/utils/url-utils.ts, src/lib/http2-client.ts, src/utils/file-utils.ts, src/app/theme-script.tsx, src/lib/circuit-breaker.ts.
What it can do
Transform developer documentation into clean Markdown
Developer documentation files → Clean Markdown files
Generate LLM-ready formatted documentation
Raw documentation content → LLM-optimized Markdown content
Create llms.txt files
Documentation source materials → llms.txt formatted files
Convert documentation to LLM-compatible format
Various documentation formats → Standardized LLM-ready text
Clean and format existing documentation
Unformatted or messy documentation → Clean, structured Markdown
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.