Requirement Adherence Boosting Data Labeling Quality Using Llms

Uber Engineering describes catching data-labeling mistakes while annotators are still working. Its Requirement Adherence system extracts individual rules from a client's instructions, allows people to add missing rules, and checks each rule separately inside the labeling tool. Formatting checks run in code; interpretation-heavy checks use reasoning models. Annotators receive suggested corrections before submitting their work. Uber reports that introducing the framework resulted in an 80% reduction in required audits. Architecture and results.