scrubbed

About

scrubbed is a single native binary that cleans raw web pages into training and evaluation text. It repairs encoding damage, extracts main content, scans for PII, and optionally runs language ID and near-duplicate detection. Replacing a chain of separate Python tools with one binary reduces dependency conflicts and deployment work. On one 400-file benchmark on an Apple M4 it used about 37 times less CPU time than a Python pipeline.