scrubbed
schancel.github.io- Category
- Developer Tools
- Rank
- No. 2455Tools index
Previous survey · No. 2351 ·
- Type
- TOOL
- Use case
- Data Processing
- Interfaces
- CLI
- Date
About
scrubbed is a single native binary that cleans raw web pages into training and evaluation text. It repairs encoding damage, extracts main content, scans for PII, and optionally runs language ID and near-duplicate detection. Replacing a chain of separate Python tools with one binary reduces dependency conflicts and deployment work. On one 400-file benchmark on an Apple M4 it used about 37 times less CPU time than a Python pipeline.
What it does
According to its site, scrubbed is a command-line binary that turns crawled web pages into clean text for model training. The site says it fixes garbled characters, strips navigation and ads, flags personal data such as emails and card numbers, and can crawl pages itself. It stops before tokenizing or training.
Stated on the product site
- License
- MIT
- Platform support
- macOS 15+ on Apple Silicon; Linux with glibc 2.36+ on x86_64 and aarch64; no native Windows build
- Install surface
- Homebrew tap, release archives, or a source build with dub
- Interface
- CLI, including a stdin/stdout JSONL stream mode
- Stated deployment scope
- Single-machine runs; distributed runs not yet tested on a representative cluster
- PII coverage
- Email, phone, card and IPv4, with report, mask and redact modes
Not stated on the site
- How does throughput compare with the Python pipeline on corpora larger than the 400-file benchmark the site reports?
- Does it hold up on multi-machine or multi-terabyte runs, which the site says have not yet been tested?
- How does encoding repair behave on text outside the Latin-1 and Windows-1252 scope the site names?
- Does PII detection extend beyond the four classes the site lists?
Written from the product site at schancel.github.io.
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.