Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models
Source
arxiv.org
Author
Vishnu Asutosh Dasu, Ashish Kundu, Gang Tan
Date
Context
Code can run and still promise something impossible. This study tests whether coding models recognize requests that depend on nonexistent software or contradict established constraints. Its benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → contains 270 deliberately unsatisfiable prompts and 91 matched solvable controls. Across twelve open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models, the authors report ungrounded code on about 60% of impossible prompts and refusals on only 27%; none of the solvable controls received a mistaken refusal. The finding exposes a gap in evaluating coding assistants: producing executable code does not establish that the requested capability exists or that the program can deliver it. Study abstract.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
articleClosed-World Resolution Against Tool Hallucination in LLM AgentsLaxmipriya Ganesh Iyer
articleAn Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming TasksChandimal Adikari, Nandika Herath
articleFrom Detection to Refusal: Safer LLMs via Circuit-Guided Weight ScalingKuan-Lin Chu, Chung-En Sun, Tsui-Wei Weng