Vibeleaderboard
← All Intel
Intel / video

HF Buckets: Upload Only What Changed with Xet

Source
youtube.com
Author
Hugging Face
Date

Context

When a large dataset changes slightly, re-uploading the whole wastes bandwidth. A Hugging Face video on Buckets, its storage feature, shows Xet, Hugging Face's storage layer, sending only the pieces of a file that changed. The presenter uploads a 191 MB NYC taxi CSV, appends 100 rows, uploads again, and reports that only the new pieces are sent.

The presenter then converts the data to Parquet, a compressed column-oriented file format, prepends 100 rows and syncs again. With PyArrow's content-defined turned on, he says, only 8.6 MB of the 52.9 MB file goes over the wire. The video calls this its one-line fix, and lists sorting, versioning and writer settings among the things to check before shipping. These figures come from one demonstration with one dataset, so savings elsewhere depend on how the data changes. The video links a written walkthrough with the commands.

Terms in this piece · Glossary
  • File — A named unit of stored data that keeps its contents while no program is using it.
  • Document chunking — Splitting source material into units that can be indexed and retrieved.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…