Vibeleaderboard
← All Intel
Intel / article

Learning from explanations

Source
stripe.dev
Author
Hangzhi Guo
Date

Context

Stripe found that adding written explanations to the fraud or not-fraud labels it trains a fraud-review model on made the model's decisions better, provided a round of label-only training came afterward. Hangzhi Guo describes the method in Learning from explanations, a follow-up to an October post in which Stripe fine-tuned a 4-billion-parameter model on reviewers' labels to estimate how likely a business account is fraudulent. In fraud review, the new approach reached 39% higher recall than label-only training, meaning it caught more of the fraudulent accounts, while flagging accounts at the same precision as Stripe's human reviewers. The new post does not say which model it used.

The obvious version failed. When Stripe simply added explanations to the training target, decisions got worse than with labels alone. Standard weighs every of the target equally, so with a 1,000-token explanation the one token carrying the decision gets about 0.1% of the training signal. Stripe's fix splits training in two: Stage 1 teaches the label followed by its explanation, and Stage 2 continues on labels alone. The explanations were written by a separate teacher model from each account record and Stripe's review guidelines, without seeing the true label, and Stripe kept only those whose verdict matched it. Human reviewers' notes were not used directly because their quality varies and not every account has one.

Two results suggest Stage 1 teaches something general about connecting evidence to a decision. More explanations helped: measured by PR-AUC, a single score for how well the model trades catching risky accounts against flagging good ones, credit decisions improved 9.8% with 15,000 Stage 1 examples and fraud decisions 3.2% with 25,000. Explanations from a different task captured most of the gain: credit PR-AUC rose from 0.578 with labels alone to 0.631 using fraud explanations, against 0.635 with credit explanations, and fraud rose from 0.710 to 0.739, against 0.747. Stripe suggests an explanation-trained model could therefore be a starting point for a new risk task before explanations for that task exist.

Label-only training had a cost: the model often stopped explaining itself, and its rubric score for explanation quality fell from 0.677 to 0.191. Stripe needs explanations to inspect assessments and investigate mistakes, so it added a third stage, on-policy , a method described by Thinking Machines and covered earlier in Intel. The model writes its own answers and the frozen Stage 1 model scores each token, steering it back toward explanations the Stage 1 model would give. Explanation quality rose to 0.835, above the Stage 1 model's, and agreement with human reviews reached 41.13, against 37.83 for the Stage 1 model, while recall at human-review precision held at 0.547 against 0.539 after Stage 2.

All of these numbers come from Stripe's own data, internal rubric and evaluation, and the idea that Stage 1 builds useful internal representations is Stripe's stated hypothesis rather than something it measured. Within those limits, the post supports a specific lesson for teams fine-tuning models on judgment tasks: explanations are a useful training signal, provided a label-only training step follows the explanation training.

Key takeaways · AI-distilled
  • Stripe found that fine-tuning on fraud labels and explanations together gave worse decisions than labels alone: SFT loss weights every token equally, so with a 1,000-token explanation the decision token gets about 0.1% of the training signal.
  • Stripe's fix is two-stage SFT: train first on labels plus explanations, then on labels alone. At human-review precision this reached 39% higher fraud-review recall than label-only SFT, while joint training fell below label-only.
  • Explanations were written by a teacher model that saw the account record and review guidelines but not the true label; Stripe kept only examples where the teacher's fraud probability agreed with that label.
  • Stripe puts the label before the explanation so the fraud score depends only on the account record. If the explanation comes first, the score can collapse toward 0 or 1 once the text commits to a verdict.
  • Label-only training cut explanation quality (rubric score 0.677 to 0.191). On-policy distillation from the Stage 1 checkpoint raised it to 0.835, above the teacher, while recall at human-review precision held at 0.547 versus 0.539.
Terms in this piece · Glossary
  • open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Recommended reads
Comments

Checking sign-in…

Loading comments…