human-in-the-loop — Requiring a person's approval at specific points in an automated process, chosen so the irreversible steps are the ones a human sees.
Why it matters
A compact catalog of production ML design patterns — like cascade models, hard negative mining, data flywheels, and business-rules layers — that give practitioners named, reusable solutions for structuring real-world AI systems instead of reinventing them per project.
Key quotes
“A key pattern when designing data pipelines is to process and aggregate raw data just once, preferably early on.”
Eugene Yan
“a study found that gpt-3.5-turbo outperformed Mechanical Turk workers for four out of five annotation tasks (relevance, topic detection, stance detection, and frame detection) on 2,382 tweets. Furthermore, the cost was less than $0.003/annotation, making it 5% of the cost of Mechanical Turk per annotation.”
Eugene Yan
“Counterintuitively, they found that models trained on hard negatives did not perform better than models trained on random negatives.”
Eugene Yan
“Through experimentation, they found that sampling hard negatives from rank 101 - 500 (in the search results) led to the best performance. Overall, blending random and hard negatives improved recall, saturating at an easy:hard ratio of 100:1.”
Eugene Yan
“Pros: One of the few sources of long-term competitive advantage. While model architecture and system design can be copied, they’re moot without the data.”
