Vibeleaderboard
← All Intel
Intel / article

Introducing Together Storage Accelerator: fast, free data caching for your GPU cluster

Source
together.ai
Date

Context

Together AI, which rents out GPU clusters for training and running AI models, has added a cache that keeps copies of a team's models and datasets on the cluster itself, included with every Together GPU Cluster at no extra cost (announcement). The problem it addresses is distance. Most teams keep their files in object storage such as Amazon S3 or Cloudflare R2, or download models from Hugging Face Hub, so each node that starts a job pulls the same files from far away. Together says those reads are slow, can be rate-limited by the source, and incur egress charges, the fees cloud providers bill for data leaving their network, while the GPUs sit idle.

Together Storage Accelerator runs on every GPU node and presents endpoints that behave like the storage a team already uses. The first time any node reads an object, the cache stores it on the cluster's NVMe drives, which are fast local flash storage, and later reads from any node are served from inside the cluster. A team switches by pointing two environment variables, `AWS_ENDPOINT_URL` and `HF_ENDPOINT`, at the cache, so existing download code does not change. Together's example is a 70 GB checkpoint needed by 200 nodes, which the cache downloads from the source once instead of 200 times.

Together's own measurements, taken on October 28 from one H100 node, show the size of the gain on repeat reads. A 2 GiB object from Cloudflare R2 took 12.7 to 15.0 seconds to download directly and 0.64 to 0.72 seconds from the warm cache, about 20 times faster, and the first read through the cache ran within about 5% of a direct download. Small files saw latency fall from 320 and 680 milliseconds to 5 and 8 milliseconds. These were single-stream reads on a preview deployment, and the 200-node figure is a count of downloads, not a timed test.

Together also notes that the same Hugging Face download ran five times faster from its us-central region than from ca-west, and says the cache's advantage is largest where the source is slowest. The strongest case is therefore a large cluster that repeatedly reads the same big files from distant storage; a team whose data already sits close to its GPUs should expect a smaller gain.

Key takeaways · AI-distilled
  • Together AI's Storage Accelerator is a cache on every GPU node in a Together cluster, in front of S3-compatible storage and Hugging Face Hub. The first read stores an object on cluster NVMe; later reads from any node stay inside the cluster. It is free with Together GPU Clusters.
  • In Together's own test on one H100 node, a 2 GiB object took 12.7 to 15.0 seconds to read directly from Cloudflare R2 and 0.64 to 0.72 seconds from a warm cache, about 20 times faster. The first, uncached read ran within about 5% of a direct download.
  • Because the cluster fetches each object from its origin only once, Together says a 70 GB checkpoint pulled by 200 nodes is downloaded once instead of 200 times. That cuts egress fees, which it puts at up to $0.09/GB from AWS, and exposure to origin rate limits.
  • Adopting it needs no new SDK: setting AWS_ENDPOINT_URL or HF_ENDPOINT to the cache endpoint keeps existing boto3, transformers and CLI code working. Together notes results depend on origin and network path, and the gain is largest where the origin is slowest.
Read the source www.together.ai
Recommended reads
Comments

Checking sign-in…

Loading comments…