Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications
- Source
- arxiv.org
- Author
- Rangeet Pan, Tyler Stennett, Divya Sankar, Bridget McGinn, Alessandro Orso, Raju Pavuluri, Saurabh Sinha, Maja Vukovic
- Date
Context
The Tangent study finds that application tests often exercise individual components while giving less to their interactions. Researchers manually labeled 2,572 test methods from 240 modules; 61.8% tested individual tools or agents in isolation. Only 7.5% addressed non-functional requirements such as security or performance. The paper reports these patterns in its sampled open-source Python applications.
The distinction matters because a test can replace an agent's model with scripted responses and still verify the surrounding software. That checks the integration under those responses, while leaving the real model's choices untested. Tangent identifies this coverage gap through test inspection, rather than measuring how often deployed agents fail.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleHow well LLM-based test generation techniques perform with newer LLM versions?Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
articleWhen Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic SystemsShihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie
Checking sign-in…
Loading comments…