Out-of-Domain Finetuning to Bootstrap Hallucination Detection

Training on a related task can help even when its immediate test score barely improves. In Eugene Yan's experiment, a model learned to detect contradictions between articles and summaries. Training first on Wikipedia summaries, then on news summaries, raised PR-AUC from 0.69 with news-only training to 0.85. This metric summarizes the tradeoff between catching inconsistent summaries and avoiding false alarms.

The Wikipedia stage alone produced little improvement on the news test. Its value emerged after the second training stage. The finding concerns one model and an article-disjoint, XSum-derived validation sample; it supports testing related data as preparation for a target task, rather than assuming that preparation must immediately improve the target score.

“It’s easy to finetune models for our specific tasks; we just need a couple hundred or a few thousand samples. However, collecting these samples is costly and time-consuming. What if, we could bootstrap our tasks with out-of-domain data?”

Eugene Yan

“To examine how out-of-domain data helps, we re-instantiate our model and finetune on Wikipedia summaries for 3 epochs before finetuning for 10 epochs on FIB. This led to PR AUC of 0.85 (right below)—a 23% improvement over finetuning on FIB alone.”

Eugene Yan

“The NLI task works like this: Given a premise sentence and a hypothesis sentence, the goal is to predict if the hypothesis is entailed by, neutral, or contradicts the premise.”

Eugene Yan

“Relative to finetuning solely on FIB, at the same threshold of 0.8, we’ve increased recall from 0.02 to 0.50 (25x) and precision from 0.67 to 0.91 (+35%).”

Eugene Yan

“It suggests that, though finetuning on USB didn’t directly improve performance on FIB, the model did learn something useful that enabled the same 10 epochs of FIB-finetuning to achieve far better results. This also means that it may be tricky to directly assess the impact of data blending.”

Eugene Yan