Vibeleaderboard
← All Intel
Intel / video

Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI

Source
youtube.com
Author
AI Engineer
Date

Context

In a large synthetic-data pipeline, fetching information about stored files can be a bottleneck separate from running the model. In its synthetic-data engineering talk, DatologyAI reports cutting S3 metadata retrieval from 9 to 11 days to about 2 hours by batching the fetches.

Metadata is information describing the stored files. The reported improvement concerns retrieving that information, not generating the whole dataset, and is one of four bottlenecks the talk covers. It illustrates why profiling the surrounding data pipeline matters: a slow preparation step can be worth fixing independently of the model-serving code.

Key takeaways · AI-distilled
  • Gaza argues that seeded rephrasing, having a model rewrite existing seed documents, produces better synthetic data than asking a model to generate text from scratch. It is the basis of DatologyAI's BeyondWeb recipe.
  • Batching S3 metadata fetches cut that step from 9 to 11 days down to about 2 hours, the first of four bottlenecks the talk covers at trillion- scale.
  • DatologyAI moved from a split Slurm and Kubernetes setup to one Ray, KubeRay and vLLM pipeline, and handles GPU failures with right-sized partitions plus checkpointing so failed work is recoverable.
  • Sweeping vLLM flags gave roughly 40% more throughput, according to the talk, alongside scheduling CPU and GPU work together across clusters.
Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…