Vibeleaderboard
← All Intel
Intel / video

How to Create an LLM Dataset | FineWeb Overview

Source
youtube.com
Author
Hugging Face
Date

Context

Hugging Face's FineWeb walkthrough shows that removing more duplicate text from a dataset does not always make it better. FineWeb is an open English web dataset built from Common Crawl snapshots, and deduplication means deleting repeated pages. The first attempt compared every snapshot against all newer ones, cutting about 36 trillion to about 4 trillion, but small test models trained on it did no better than on the undeduplicated data.

The presenter explains the cause: in an older snapshot, a trained on the removed text outperformed one trained on what was kept, which was mostly templates, navigation and ads. The team switched to deduplicating each snapshot only against itself, giving a roughly 15 trillion token dataset that matched the earlier RefinedWeb dataset. The evidence is small models scored on aggregate benchmarks, so it shows what worked here, not a general rule. The video recaps a project from a couple of years earlier.

Terms in this piece · Glossary
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Large language model — A model trained on large amounts of data to process and generate language.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…