Fact-Checking at Scale: Building a More Accurate and Efficient Synopsis Factuality System

Netflix checks the short synopses it writes for every show and episode with an automated fact-checker, and a post on its tech blog describes how the team made that checker cheaper while catching more bad synopses. The existing checker hands a reasoning model everything Netflix knows about a title, often more than 100,000 tokens of metadata and text, along with a factuality rubric, and asks for a verdict. It runs that same call five times and fails the synopsis if any run objects. The new design spends model calls differently: on specialized checks, and only on the synopses where the model is unsure.

Factuality here means more than avoiding false statements. Using Bridgerton as the example, the post separates synopses that are plainly wrong, such as one describing a modern-day crime drama, from synopses in which every sentence is true but the overall picture misleads: one that centers the show on the gossip writer Lady Whistledown, or one that implies Daphne and the Duke fall in love right away when their relationship begins as a fake arrangement. When the team studied the old checker, the five repeated runs agreed with each other on the blatant cases and disagreed mostly on the subtle ones. Repetition also cannot fix a reasoning mistake that every run shares. The model sometimes failed synopses over nitpicks and sometimes talked itself out of a real problem by deciding it did not make the synopsis false.

The replacement is a cascade. Every synopsis goes to two judges: a literal judge that asks whether each statement is true, and an implicature judge that asks whether the synopsis would give a false picture to someone who has not watched the show. Each returns a verdict, its reasoning and a confidence level, and confident verdicts stand. The judges are also asked to list their "concessions": assumptions they made, doubts the evidence could not settle, and contradictions they noticed but waved away. For uncertain verdicts, a router model rewrites each concession as a standalone claim and sends it either to a web judge, for outside facts such as a critic's quote, or to a narrower recheck against the title's source material. This second stage can flip a fail to a pass if every concession checks out, or a pass to a fail if any one is contradicted.

On a benchmark of synopses that Netflix's annotators labeled by hand, the cascade caught 86% of the show-level synopses humans judged non-factual, against 77% for the old checker, and 92% against 86% for episode synopses, where both versions were given a summary of the whole show. At the show level it averaged 3.38 model calls per synopsis instead of five, and the cost of running the whole dataset fell from $1,573.46 to $748.34 in the post's table, the 52.4% saving it reports. Its written reasons also matched the annotators' reasons more closely, which Netflix summarizes as roughly 82% overlap, up from about 69%; that overlap was scored by two language models, Opus 4.5 and Gemini 3.1 Pro, comparing each explanation with the human one. Netflix wants better-aligned reasons because it hopes to use them later to correct faulty synopses automatically.

The gains have costs that the results table makes visible. The median time to a verdict nearly doubled at the show level, which Netflix still considers acceptable. The checker also remains strict: of the show-level synopses that annotators considered factual, the cascade passed 58%, slightly fewer than the old checker's 61%, so a large share of accurate synopses still get flagged. The post does not say what happens to a flagged synopsis after that, and all of the figures come from Netflix's own benchmark.