Do not grade a GPT agent with a GPT judge

LangChain posted a short clip in which a speaker, credited in the post as @HeggieConnor of @unifygtm, states a rule for testing AI agents: if the agent runs on a GPT model, the model judging its work must come from a different family. An LLM judge is a second language model asked to grade the first model's output, a common way to check agent runs when people cannot review every one.

In the clip, the speaker gives a concrete case: an agent running on GPT-5.4 gets an Anthropic model as its judge. The speaker's reason is that models from the same family have "overlapping distributions," meaning they learned similar patterns, and when they review each other the result is mode collapse, which the speaker compares to groupthink in humans. The speaker wants the judge to be "almost adversarial," drawn from a completely different model.

In practical terms, a judge that shares the agent's habits may miss the same errors, so a passing grade tells the team less than it seems to. Amazon researchers made a related argument about panels of judges: judges that share a model family or training lineage can repeat the same mistake, so their agreement looks like stronger evidence than it is. The clip itself offers no measurement of how often a same-family judge misses problems; it is one team's engineering rule, stated from experience.