Intel / article

When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations

Source: arxiv.org
Author: Jinhan Kim, Samuele Pasini, Paolo Tonella
Date: Published Jul 30, 2026

Context

A document- chatbot needs testing when its reference material changes, not only against one frozen collection. This research deliberately alters documents or retrieved passages, then checks whether the system's responses obey the expected relationship between the original and changed cases. The method, called metamorphic testing, uses 11 kinds of alteration. Across five datasets and more than 28,000 modified cases, the authors report consistency violations of 4.9–10.2%. These rates describe failures under the chosen alterations, rather than overall answer accuracy. The method provides a way to investigate whether knowledge updates or added noise disrupt a previously evaluated system. Research abstract.

Key takeaways · AI-distilled

Metamorphic testing mutates the corpus and asks whether the answer changed the way it should have. No answer key is needed, which is exactly why it survives a corpus that keeps changing underneath you.
The 11 mutation operators split across two levels: the retrieval index before , and the retrieved after. That separates 'the wrong document was found' from 'the right document arrived mangled.'
Across five datasets and over 28,000 mutants, 4.9% to 10.2% of mutations produced an inconsistent answer. That failure rate is invisible to any evaluation run against a fixed snapshot.
The metamorphic oracle caught these faults at F1 of 0.927 to 1.000 while the best RAGAS metric reached 0.570. Standard metrics barely detect update-induced breakage.

Terms in this piece · Glossary

grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
chunking — Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data.”

“An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%.”

“In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570.”

Read the source arxiv.org