crawlkit

Why it's ranked here

The project has a disciplined boundary and unusually broad practical coverage. It combines careful SQLite handling, portable snapshots, encrypted backups, vector search, scheduling, and terminal interfaces without absorbing provider-specific policy. The documented compatibility gates make that breadth more credible.

What's good

The SQLite layer sets conservative connection defaults, supports read-only access, checks schema versions, and tightens database permissions. Snapshots support fingerprints and incremental imports. Backups encrypt compressed shards and validate hashes during restoration. Search covers multiple embedding endpoints, vector validation, exact cosine search, top-k selection, and result fusion.

Tradeoffs

It is infrastructure, not a complete crawler. Provider APIs, authentication, schemas, privacy filters, ranking, and command design remain downstream responsibilities. Hosted archive deployment, authorization policy, database migrations, and secrets also live elsewhere. Adoption may require adapters because existing applications retain different synchronization table shapes.

How to use it well

Choose it when building several Go crawler applications that should share archive mechanics without surrendering their existing data or command contracts. Adopt focused packages incrementally, keep privacy decisions beside provider code, and verify compatibility against old fixtures. Use another service or application layer for provider ingestion, authentication, hosted deployment, and product-specific search behavior.

Observed

License

MIT

Primary language

Go

Install surface

Go module packages via go get, plus an optional crawlctl binary via go install

Interfaces

Go library, command-line controller, terminal browser, and provider-neutral remote HTTP client contract

Go requirement

Go 1.26.5 or newer

Scheduling platforms

launchd on macOS, systemd user units on Linux, Task Scheduler on Windows, with cron rendering as fallback