Commit-first LLM judging inherits the judge's own errors

Making an AI evaluator solve a task before grading another answer can protect it from adopting that answer's mistakes. But the evaluator's own solution can also be wrong. A controlled coding study tested this commit-first approach by fixing the judge's solution before comparing candidate programs against it.

On a task that combined overlapping intervals, the change eliminated the study's cases of highly scored programs failing hidden tests. On a duration-parsing task, it increased those cases in both runs: the judge's own parser accepted malformed inputs. The result makes task-specific competence central to evaluator design. An independently produced reference answer is useful only to the extent that it is correct.