← All IntelClip / AI ToolsShort randomized annotation sessions calibrate the judges and seed the next model
From Evaling Video Slop — Maor Bril, Character.ai · ≈18:28
A lightweight, repeatable human-taste process, with axes randomized per annotator so no one rates the same video on ten dimensions, and the labels double as training data for the next judge version.
What’s in it
- A lightweight, repeatable human-taste process, with axes randomized per annotator so no one rates the same video on ten dimensions, and the labels double as training data for the next judge version.
Clip transcript
judges? >> Yeah. So, So, So, So, this is actually solved at first at the the the Judge Judy part, where every report it will generate a human can go and annotate it. And we actually we we do that. We we will periodically have sessions where everyone spends 10 to 15 minutes just annotating videos. And and that usually happens on multiple axes. I won't ask everyone to annotate the same video on on 10 different things. It'll It'll It'll be random. And I use the data to to calibrate the the AI judges. And and the results from that is actually being served as as a data set for for training for the next version of of that model. So, it's a process that does take a little bit of time and and hopefully and and it does evolve over time. Uh but it's not immediate you know, because also taste is very subjective and and things that are great for me, you know, some that I think are fantastics some people that come and say are you sure they're great because you know So, yeah, it's it's it's a process and and and and I use the human feedback to calibrate the models all the time.
Comments
Checking sign-in…
Loading comments…