Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Shows where synthetic data now fits in each stage of the LLM pipeline and why controlled generation can beat filtering the web, informing small-model and data decisions.

• pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
• LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

• videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
• videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
• videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
• videoExo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research

• article[AINews] 10% worse, 100x cheaper, 10000x faster: Why Simulation is taking overwww.latent.space
• articleWhen Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey ResponsesZihan Chen, Di Zhu, Lei Nico Zheng
• clipMAI Thinking 1: no synthetic data, but web text still collapses to 15%AI Engineer

Checking sign-in…

Loading comments…