← All IntelClip / EducationWhere the run actually peaked, and what overfitting looked like
From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈51:08
An empirical trace — nonsense at 200 steps, decent by 800, rising validation loss marking the turn, and models becoming less creative past the peak.
What’s in it
- An empirical trace — nonsense at 200 steps, decent by 800, rising validation loss marking the turn, and models becoming less creative past the peak.
Clip transcript
but it will no longer start be getting better at it. Uh this is like an example at 200 steps when I was testing this. Uh it was just producing like complete nonsense. Uh and the val loss was around 3.5. Uh then at about 800 steps, it started producing like decent things. Uh still not not not great things, but it was it was starting to get there. And at 1,000 steps, it got better. And there was one point that the val loss actually started increasing instead of decreasing, and that's where we know the model overfit. So at around 2,400 steps is where the that was the optimal performance of this model. And then if we kept going, the the performance was actually maybe not decreasing, but the models are becoming less creative. So, that's one of the things to to keep in mind. Now, val loss is like not the best metric like for if you're actually being serious about training LLMs, usually you might have like some benchmarks that are running as part of your training and you can see like if the benchmarks are getting worse or not. Uh but for us that's like a very easy and cheap way of like understanding how the model is doing.
Comments
Checking sign-in…
Loading comments…