Crawlee
apify.com- Category
- Developer Tools
- Rank
- No. 743Tools index
Previous survey · No. 718 ·
- Pricing
- Open Source
- Type
- TOOL
- Interfaces
- CLI · SDK · Desktop
- Builder
- @apify
- GitHub
- 26.1k stars
- Latest release
- v4.0.0-rc.1
- Date
About
Crawlee is an open-source Node.js library that unifies HTTP and headless-browser crawling (via Playwright or Puppeteer) under one interface, with built-in anti-blocking, proxy rotation, persistent request queues, and pluggable storage for building reliable web scrapers.
What it does
A Node.js toolkit for building web crawlers that can fetch pages directly over HTTP or drive a real browser, using the same code either way. It manages the parts that make scraping tedious at scale: a queue of pages still to visit, saved results, work spread across concurrent jobs, and rotating IP addresses and browser identities so sites are less likely to notice.
Why it's ranked here
This is a mature, well-factored project, not a thin wrapper. It's Apache 2.0, split into a dozen focused packages sharing one core, and its test suite includes dockerized integration services rather than mocks alone. The anti-blocking design goes past a checkbox: sessions bind an IP, cookie jar, and fingerprint together and rotate them as a unit, with a dedicated TLS-impersonation client and a stealth browser build for tougher walls like Cloudflare.
What's good
The single interface across HTTP and browser crawling is the real win: swap engines without rewriting request handling. Packages are split so you install only what you need, and a CLI scaffolds a working project in one command. Session handling ties IP, cookies, and browser fingerprint together as one rotating identity, which is a more coherent anti-blocking model than juggling those separately. Storage location, retries, routing, and error handling are all configurable rather than fixed.
Tradeoffs
Browser automation isn't bundled: Playwright and Puppeteer are separate installs, added deliberately to keep the base package small, but that means an extra setup step before browser crawling works. Default storage writes to a local folder, which needs to be overridden before deploying anywhere without persistent disk. The project ships as a Node.js and TypeScript library only; teams standardized on Python get a separate, differently maintained project.
How to use it well
Good fit for Node or TypeScript teams that need both light scraping and full browser rendering without adopting two frameworks. Start with the plain HTTP crawler for speed when pages don't need JavaScript, and switch to a browser-based one only when content requires rendering, since the interface stays the same either way. Configure session and fingerprint options up front if the target site fingerprints bots, and point storage at a real database or bucket before running this in production rather than trusting the local default.
Technical notes+
The package split shows in the source tree: packages/core/src/index.ts holds the shared primitives (configuration, request, storages), packages/basic-crawler/src/index.ts adds autoscaling, session pooling, and routing on top of it, and packages/browser-crawler/src/index.ts layers browser launch and control before packages/playwright-crawler/src/index.ts and packages/cheerio-crawler/src/index.ts specialize into the two crawl modes. packages/browser-pool/src/index.ts is a separate module that manages Puppeteer and Playwright browser instances as a pool, independent of the crawler classes. The CLI entry at packages/crawlee/src/cli.ts is a thin shim that defers to a local install via import-local before falling back to @crawlee/cli. packages/utils/src/index.ts exposes standalone helpers (HTML-to-text conversion, sitemap and robots.txt parsing, URL extraction) usable outside the crawler classes entirely.
Observed
- License
- Apache License 2.0
- Language
- TypeScript
- Package
- Published to npm as the crawlee package, plus scoped sub-packages such as @crawlee/core and @crawlee/playwright
- Interfaces
- Library API plus a CLI (crawlee create) for scaffolding new projects
- Runtime requirement
- Node.js 16 or higher
- Browser engines
- Chromium, Firefox and Webkit via Playwright; Chromium and Chrome via Puppeteer, both optional peer installs rather than bundled dependencies
- Repository structure
- Monorepo of focused packages (core, basic-crawler, browser-crawler, playwright-crawler, cheerio-crawler, browser-pool, utils, cli) built with pnpm workspaces and Turborepo
- Test tooling
- Vitest test suite, with a separate integration project that runs against Docker-based browserless and httpbin services
- Default storage
- Local ./storage directory, overridable via the CRAWLEE_STORAGE_DIR environment variable
- HTTP parsing
- Cheerio and JSDOM used for HTML parsing in non-browser crawl mode
Read from README.md, package.json, packages/core/src/index.ts, packages/basic-crawler/src/index.ts, packages/browser-crawler/src/index.ts, packages/playwright-crawler/src/index.ts, packages/cheerio-crawler/src/index.ts, packages/browser-pool/src/index.ts, packages/crawlee/src/cli.ts, packages/utils/src/index.ts, docs/introduction/index.mdx, docs/quick-start/index.mdx, docs/guides/avoid_blocking.mdx, CONTRIBUTING.md, LICENSE.md.
What it can do
Crawl websites via HTTP requests
URLs → Scraped web data
Crawl websites using headless browsers (Playwright or Puppeteer)
URLs → Scraped web data
Rotate proxies to avoid blocking
Proxy list → Requests routed through different proxies
Persist request queues across crawl sessions
Crawl requests → Stored request queue
Store scraped data via pluggable storage backends
Scraped data → Stored data records
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.