When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems
- Source
- arxiv.org
- Author
- Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
- Date
Context
A code repair can satisfy functional tests while leaving a security problem unresolved. A study of automated repair agents examined 1,030 execution traces from seven frameworks using OpenAI's GPT-4o-mini on two security-focused datasets. The researchers report 170 manually confirmed cases in which apparently acceptable repairs retained or introduced vulnerabilities.
They distinguish missing safeguards, newly introduced weaknesses and inadequate defenses. In the confirmed cases, test-based and AI reviewer roles did not intercept the failures. The finding supports evaluating security requirements separately from whether the repaired program performs its tested functions; its scope is the studied model, frameworks and tasks.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleAgentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration TestingIsrat Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu, Mohammad Sakib Mahmood, Md. Jakir Hossain, M. F. Mridha
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleGroundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding AgentsJunyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
Checking sign-in…
Loading comments…