When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations
Source
Jinhan Kim, Samuele Pasini, Paolo Tonella
Author
Jinhan Kim, Samuele Pasini, Paolo Tonella
Date
Key takeaways · AI-distilled
Metamorphic testing mutates the corpus and asks whether the answer changed the way it should have. No answer key is needed, which is exactly why it survives a corpus that keeps changing underneath you.
The 11 mutation operators split across two levels: the retrieval index before chunkingSplitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.Full definition →, and the retrieved context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → after. That separates 'the wrong document was found' from 'the right document arrived mangled.'
Across five datasets and over 28,000 mutants, 4.9% to 10.2% of mutations produced an inconsistent answer. That failure rate is invisible to any evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → run against a fixed snapshot.
The metamorphic oracle caught these faults at F1 of 0.927 to 1.000 while the best RAGAS metric reached 0.570. Standard RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → metrics barely detect update-induced breakage.
Terms in this piece · Glossary
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
chunking — Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Your RAG evals pass on yesterday's snapshot while a routine document update quietly breaks answers.
Key quotes
“However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data.”
“An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%.”
“In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570.”
articleUniversal Pathologies, Conditional Consequences: A Triple-Robustness Analysis of RAG for Multi-Hop TraceabilityMeftun Akarsu, Burak Ozdemir
articleTemporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub FixesNeeraj Yadav
articleRepository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language ModelsDibyendu Brinto Bose, Jiawei Qin, Chris Brown