Intel / video

Evaling Video Slop — Maor Bril, Character.ai

Source: youtube.com
Author: AI Engineer
Date: Published Jul 25, 2026

Context

An automated video judge can reward visual polish while missing implausible movement. In this Character.AI presentation, the speaker describes an early evaluator that gave high marks to a nearly still shot for camerawork and to hovering figures for physics, because its training data taught it to recognize the look of coherent or artificial video instead of the intended quality criteria.

The revision paired real footage with AI-generated footage, with both sides encoded consistently and annotated by the same method so the judge could not simply learn to tell AI video apart. Human annotators also the judges on criteria such as storytelling, pacing and physical plausibility. The approach makes the training examples part of the problem: a judge needs examples that distinguish the quality people want from convenient visual shortcuts.

Terms in this piece · Glossary

calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.

Key quotes

“the hard part was never how to make video. The hard part was how do we generate um good enough video and how do we judge if the video is good enough?”
Maor Bril

“If you think about what is video, video is a storytelling medium.”
Maor Bril

“But if I show you two videos and I'll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A, right?”
Maor Bril

“Now, we shipped V1, and it was so wrong. It It was wrong, but it was wrong in a very confident way.”
Maor Bril

“it scored the vibe as opposed to the the the axes”
Maor Bril

Read the source www.youtube.com