More Design Patterns For Machine Learning Systems

A recommendation system can reserve expensive scoring for a shortlist. In a machine-learning patterns article, Eugene Yan describes a cascade: a fast retrieval stage selects candidates from a large collection, then a slower ranking stage examines that smaller set more carefully.

The stages have different jobs. Yan describes retrieval as optimizing for recall and speed and ranking as optimizing for precision. That division concentrates computation, but also makes the stages dependent: an item excluded from the shortlist is never seen by the ranker.

“A key pattern when designing data pipelines is to process and aggregate raw data just once, preferably early on.”

Eugene Yan

“a study found that gpt-3.5-turbo outperformed Mechanical Turk workers for four out of five annotation tasks (relevance, topic detection, stance detection, and frame detection) on 2,382 tweets. Furthermore, the cost was less than $0.003/annotation, making it 5% of the cost of Mechanical Turk per annotation.”

Eugene Yan

“Counterintuitively, they found that models trained on hard negatives did not perform better than models trained on random negatives.”

Eugene Yan

“Through experimentation, they found that sampling hard negatives from rank 101 - 500 (in the search results) led to the best performance. Overall, blending random and hard negatives improved recall, saturating at an easy:hard ratio of 100:1.”

Eugene Yan

“Pros: One of the few sources of long-term competitive advantage. While model architecture and system design can be copied, they’re moot without the data.”

Eugene Yan