Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams
Source
arxiv.org
Author
Kevin Delcourt, Meriem Ben Chaaben, Abdelhamid Rouatbi, Luciano Marchezan, Houari Sahraoui
Date
Why it matters
If you rely on an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge, mutation testing gives you a way to measure whether it actually detects defects instead of trusting spot-checks: inject known faults, score detection, compare to human validation.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleHype Meets Reality: Large Language Models as Mutators in Search-based Automated Program Repair of Simulink-Stateflow ModelsAyesha Irshad, Pablo Valle, Jon Ayerdi, Aitor Arrieta
articleAuditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL PipelineHaowei Liu, Hsin-Tai Wu, Yi Fang