How Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific Programming
Source
Gabrielle O'Brien, Reed Milewicz, Nasir Eisty
Author
Gabrielle O'Brien, Reed Milewicz, Nasir Eisty
Date
Key takeaways · AI-distilled
Drawn from 527 free-text responses to a 2025 survey, mostly researchers at U.S. universities, each describing one real task, how they used an AI tool for it, and how they checked the result.
Usage concentrated in five tasks - data handling, visualization, debugging, mathematical/scientific computing, and statistical analysis - and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → strategies barely varied with how experienced the programmer was.
Confidence, not method, tracked experience: less experienced programmers trusted the AI's output more than their own judgment, while experienced programmers trusted their own judgment more, the reverse pattern.
Evaluation confidence correlated most strongly with confidence in the tool and in oneself, not with which verification strategy (running the code, tests, peer review) was actually used.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Shows that even technically experienced researchers verify AI-generated code mostly by running it once, rarely with tests or review, and that confidence tracks experience more than accuracy, a gap worth addressing in any AI-assisted coding workflow.
articleA Study of the Reliability of Agentic AI-Generated ProgramsAyesha Shafique, Barton P. MIller, Elisa R. Heymann
articleThe Vibe Shift in Software Engineering: Evaluating AI-Led Conversational Programming for Performance, Cognition, and Responsible AdoptionSales G. Aribe Jr., Louie Jay S. Labastida
articleVibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art ReviewDominik L. Michels, Mutaz Abu Ghazaleh, Francois Lazzari, Nabil Kassem, Jonathan Klein