Shows a production pattern for LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition → review: tune judge prompts against a calibration set of human-approved examples, keep humans on edge cases, and analyze failure patterns instead of only scoring.
Key takeaways · AI-distilled
Scale says the labs it works with accept 97% of its data on the first pass, backed by three review levels: task-level checks of each annotation, dataset-level evaluation of training value, and contributor-level assessment of annotators.
Scale's Archie system uses AI agents as judges of each annotation against project rules, with human experts on edge cases. Scale requires it to match expert human reviewers on pre-delivery checks before deployment.
Archie tunes its judge prompts through auto-hillclimbing, iteratively testing them against a calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → dataset of ideal examples, and generates reports that summarize common failure modes rather than only scoring.
Scale's dataset-level framework combines diversity, relevance, difficulty and generalization metrics, including reward-hacking detection, into a composite score meant to predict a dataset's impact on a model before training.
Contributor checks estimate the likelihood a response was LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →-generated, flag anomalous interaction patterns and filter gibberish, while a fit model matches annotators to projects and moves them to harder work as they show ability.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
