Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

Meta says its system for writing "catching" tests with large language models is no longer a research prototype but part of its infrastructure, and that in the first half of 2026 it played a key role in catching 74 serious privacy bugs before they reached production. A catching test is generated for one specific proposed change and is meant to fail, showing a concrete difference in how the code behaves after the change. That distinguishes it from AI code review, which reads a change and judges it the way a human reviewer would. According to Meta's engineering post, running tests costs more computation but catches classes of bugs that review can miss.

The system runs overnight on spare capacity against the previous day's changes that a risk model flags as likely to cause severe regressions. Two workflows generate tests. One treats every change as if it were a mutant, the deliberately planted bug used in mutation testing, and writes tests that tell the new code apart from the old. The other has a model infer what the change is meant to do, list the ways that intent could be implemented wrongly, build a mutant for each risk, and generate tests that catch those mutants, then runs the tests on the real change. Any failing test is only a candidate, which Meta calls a weak catch, because the failure may simply reflect behavior the engineer changed on purpose. Meta filters them with an ensemble of language models acting as judges and a deterministic rule-based checker it calls RubFake, then contacts engineers only about candidates that rank high enough. Meta calls this human-on-the-loop: engineers set criteria and review outcomes in aggregate, and when contacted they mainly confirm whether a behavior change was intended.

The reported results are Meta's own. It says 589 engineers, 64% of those it contacted, have engaged with the findings, and that they report 60% of findings as true positives. It also says privacy incidents classed as serious fell 20% over the same six months, though the post's text does not separate the system's contribution from other causes or give the total number of changes tested. For scale, Meta's January paper on the initial deployment reported 41 candidate catches sent to engineers, of which 8 were confirmed bugs and 4 would have caused serious failures.