← All IntelClip / AI ToolsDisagreement as signal, not noise, in human QA
From Ending AI Slop — Thais Castello Branco, Taste Labs · ≈13:57
“I think the key is understanding is that disagreement something that's actually a flaw in the data meaning are they for example disagreeing on something that they should be agreeing on such as alignment”
“alignment is something that's pretty objective So it would be kind of odd to see experts disagreeing on that front.”
“if they're disagreeing on things like sty style or um aesthetics that is not necessarily bad data that's actually good data.”
What’s in it
- Explains how to design human QA loops for preference datasets
- Shows why evaluator disagreement isn't always a red flag
- Breaks down objective vs subjective criteria in data quality checks
Clip transcript
characteristics about it that will uh correlate highly let's say with with valuable data and same with human QA. I think by the way human QA is tricky here because there's two sides to it. There is a side of for example let's say you're collecting uh a bunch of preference data about slide design. Um you could have human QA be like okay is this again high quality data is the following the specs which most people would agree on but suddenly if you ask for expert consensus and you try to have another expert see if they agree with the initial designers like votes you might start seeing some disagreement there and you I think the key is understanding is that disagreement something that's actually a flaw in the data meaning are they for example disagreeing on something that they should be agreeing on such as alignment like alignment is something that's pretty objective So it would be kind of odd to see experts disagreeing on that front. But suddenly if they're disagreeing on things like sty style or um aesthetics that is not necessarily bad data that's actually good data. It shows you that there is a distinction for what people like. And so uh this almost like analysis of how do you run human QA in a way that both screens for kind of the fundamentals but then when you use consensus I think in an intentional way is another big big piece of this. Um and then obviously kind of
Comments
Sign in to comment.
Loading comments…