Adapting Dolma for Thai: what broke and what held up

Mangosteen shows which parts of an English-built data-cleaning pipeline carry over to another language and which parts quietly break. Ai2's Dolma toolkit cleans web text in four stages: keep only the target language, drop low-quality pages by rule, remove duplicates, and filter unwanted content. The Thai team behind the Mangosteen paper kept that structure and the parts that worked unchanged, such as removing duplicate URLs and whole duplicate documents, and rewrote most of the language-specific checks.

Many of those rewrites follow from how Thai is written. The authors found that standard language detectors were unreliable on Thai, so they kept only documents in which at least half of the characters are Thai. They dropped a rule that discards lines without ending punctuation, because Thai sentences do not normally end with a period. Duplicate removal by paragraph was ineffective because line breaks are not reliable paragraph boundaries in Thai; Ai2's blog post says sentence- and paragraph-level deduplication removed almost all of the team's data. They also raised the minimum page length from 50 to 200 words after finding that short Thai pages were mostly low quality or came from pornography and gambling sites, added a rule against "read more" snippet pages, and trained Thai-specific filters for adult and gambling content.

"Less data" in Ai2's thread needs one qualification. In the paper's main tests, every version of the data trained a small GPT-2 model on the same 10 billion tokens, so the comparison is about which text fills that budget, not about training on fewer tokens. On raw Common Crawl, the full pipeline cut 202 million documents to about 25 million and raised a Thai text-generation score from about 3 to about 11. On FineWeb2, a dataset that was already cleaned, it roughly halved the documents with mixed results: better on one benchmark suite, slightly worse on the other.

The authors then used the 47-billion-token corpus to continue training an 8-billion-parameter SEA-LION model. With the same fine-tuning as three comparison models, it had the highest average on the SEA-HELM suite, but on the Thai LLM Leaderboard it led on generation tasks and scored below two of the three comparison models on comprehension tasks, which the authors attribute to lost world knowledge. The paper gives its lead as about four points in the abstract and about two in the conclusion, and says its compute budget allowed effectively one full training run. The paper dates from July 2025; Ai2's August 2026 post presents it as an example of what an open curation toolkit lets other teams do.