Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

Training a small language model makes the difference between learning and memorization easier to inspect. A researcher at ElevenLabs uses this workshop to build a model from random starting weights and train it to predict the next character in Shakespeare text.

The training process adjusts the model using one portion of the text and checks its predictions on a separate, held-out portion. A falling training error shows improvement on the material used for those adjustments. If held-out performance stops improving while training performance continues to improve, that can expose overfitting: increasingly fitting the training examples without a corresponding gain on unseen material. The exercise makes this distinction concrete without requiring a frontier-scale model.

“Okay, we need to train this new TTS model and we're going to spend maybe 6 months thinking about the tokenizer, and then we're going to spend 2 months on the architecture.”

Angelos Perivolaropoulos

“you don't necessarily need to know in a very deep level how transformers work to to be able to train something like this”

Angelos Perivolaropoulos

“I remember when OpenAI was about to release uh GPT-2 and they were saying we're not going to release it because it's too dangerous for for humanity”

Angelos Perivolaropoulos

“if this data set has like even some small issues it can literally make or break your model”

Angelos Perivolaropoulos

“you can't just go on like Reddit and just get random posts. You're not going to learn how to think this way for sure.”

Angelos Perivolaropoulos