Adversarial validation in one move: label dataset A as 0 and dataset B as 1, then fit a binary classifier to tell them apart. If it can — say AUC at or above 0.60 — the two sets genuinely differ, and that difference is your drift.
The payoff is the explanation, not the alarm. Use an interpretable model like logistic regression and read feature importances to see which tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → drove the split; the author's tool surfaced tags like UI-FORMAT and traced a bug to a changed prompt template.
A negative result proves nothing. Failing to detect drift only means your chosen features and model could not find it, so treat the check as a cheap alarm rather than a clean bill of health.
Token features catch template, schema, and formatting drift. Add embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → as features to catch semantic drift, plus hand-built ones like conversation turn count or message length to catch changes in shape.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
Why it matters
A cheap, infrastructure-free way to catch the silent killer of evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → validity — your eval set no longer matching what production actually sends your model.
Key quotes
“For years, I’ve relied on a straightforward method to identify sudden changes in model inputs or training data, known as “drift.””
“Your data for evaluations are materially different from the inputs your model receives in production, causing your evaluations to be misleading.”
“This method is so simple that it might seem uncool. You aren’t going to impress any data scientists. Despite this, it’s too valuable to ignore.”
“If the classifier demonstrates sufficient predictive power (ex: AUC >=0.60), we know there is drift.”
“If this process doesn’t detect drift, it doesn’t mean there isn’t drift. It just means that we couldn’t detect it with the model and features we used.”
