HF Buckets: Upload Only What Changed with Xet
- Source
- youtube.com
- Author
- Hugging Face
- Date
Context
When a large dataset changes slightly, re-uploading the whole wastes bandwidth. A Hugging Face video on Buckets, its storage feature, shows Xet, Hugging Face's storage layer, sending only the pieces of a file that changed. The presenter uploads a 191 MB NYC taxi CSV, appends 100 rows, uploads again, and reports that only the new pieces are sent.
The presenter then converts the data to Parquet, a compressed column-oriented file format, prepends 100 rows and syncs again. With PyArrow's content-defined turned on, he says, only 8.6 MB of the 52.9 MB file goes over the wire. The video calls this its one-line fix, and lists sorting, versioning and writer settings among the things to check before shipping. These figures come from one demonstration with one dataset, so savings elsewhere depend on how the data changes. The video links a written walkthrough with the commands.
- File — A named unit of stored data that keeps its contents while no program is using it.
- Document chunking — Splitting source material into units that can be indexed and retrieved.
Checking sign-in…
Loading comments…






