Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale
Source
engineering.fb.com
Date
Why it matters
Shows LLM-generated tests run per code change in a very large codebase, with fewer false positives and less load than traditional testing. Gives teams a reference design for gating changes with generated tests.
Key takeaways · AI-distilled
Meta runs two test-generation workflows in tandem. A 'dodgy diff' workflow treats every change as a mutant and writes tests that tell it apart from its parent. An 'intent-aware' workflow infers the diff's intent, lists ways it could go wrong, and builds mutants from those risks.
To sort real bugs from false positives, Meta pairs an LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition → ensemble of several frontier models with RubFake, a rule-based assessor. Meta notes the LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judges are richer but non-deterministic, while RubFake is deterministic; together they rank candidates.
Meta's system runs overnight on spare capacity and only targets the previous day's diffs flagged as risky by a mechanism similar to its Diff Risk Score, trained on past code changes, so generation effort goes where severe regressions are most likely.
Meta says 589 engineers, 64% of those it contacted, engaged with the tests and reported a 60% true-positive rate. In the first half of 2026 the system helped catch 74 serious privacy bugs before landing and cut privacy SEVs by 20%.
Unlike LLM code review, which mimics a human reviewer, Meta's approach produces executable tests showing a concrete behavior change. Meta says this costs more compute but catches bug classes review can miss, and engineers mostly just confirm whether the change was intended.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
articleAgentic Testing and the Bugs Coding Agents Actually ShipAndrew Ng
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleHow well LLM-based test generation techniques perform with newer LLM versions?Michael Konstantinou, Renzo Degiovanni, Mike Papadakis