Vibeleaderboard
Index / tool
Visit schancel.github.io
Category
Developer Tools
Rank
No. 2455Tools index

Previous survey · No. 2351 ·

Type
TOOL
Use case
Data Processing
Interfaces
CLI
Date

About

scrubbed is a single native binary that cleans raw web pages into training and evaluation text. It repairs encoding damage, extracts main content, scans for PII, and optionally runs language ID and near-duplicate detection. Replacing a chain of separate Python tools with one binary reduces dependency conflicts and deployment work. On one 400-file benchmark on an Apple M4 it used about 37 times less CPU time than a Python pipeline.

What it does

According to its site, scrubbed is a command-line binary that turns crawled web pages into clean text for model training. The site says it fixes garbled characters, strips navigation and ads, flags personal data such as emails and card numbers, and can crawl pages itself. It stops before tokenizing or training.

Stated on the product site

License
MIT
Platform support
macOS 15+ on Apple Silicon; Linux with glibc 2.36+ on x86_64 and aarch64; no native Windows build
Install surface
Homebrew tap, release archives, or a source build with dub
Interface
CLI, including a stdin/stdout JSONL stream mode
Stated deployment scope
Single-machine runs; distributed runs not yet tested on a representative cluster
PII coverage
Email, phone, card and IPv4, with report, mask and redact modes

Not stated on the site

  • How does throughput compare with the Python pipeline on corpora larger than the 400-file benchmark the site reports?
  • Does it hold up on multi-machine or multi-terabyte runs, which the site says have not yet been tested?
  • How does encoding repair behave on text outside the Latin-1 and Windows-1252 scope the site names?
  • Does PII detection extend beyond the four classes the site lists?

Written from the product site at schancel.github.io.

Tags

data-cleaningllm-trainingpiimojibakecliweb-scraping

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.