Intel / article

How Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific Programming

Source: arxiv.org
Author: Gabrielle O'Brien, Reed Milewicz, Nasir Eisty
Date: Published Sep 21, 2026

Context

Researchers' confidence in AI-generated code was not associated with the checking strategies they reported in a study of 527 survey responses. Each response described one scientific-programming task, how AI was used and how the result was assessed. More than half mentioned running the code; automated tests and another person's review were uncommon.

In the reported analysis, confidence was more strongly associated with trust in the tool and in one's own programming ability. Less experienced programmers tended to trust AI more than themselves, with the opposite pattern among experienced programmers. These are self-reported practices, not measurements of code correctness. The finding identifies a gap between feeling able to judge an answer and describing a shared, testable validation process.

Key takeaways · AI-distilled

Drawn from 527 free-text responses to a 2025 survey, mostly researchers at U.S. universities, each describing one real task, how they used an AI tool for it, and how they checked the result.
Usage concentrated in five tasks - data handling, visualization, debugging, mathematical/scientific computing, and statistical analysis - and strategies barely varied with how experienced the programmer was.
Confidence, not method, tracked experience: less experienced programmers trusted the AI's output more than their own judgment, while experienced programmers trusted their own judgment more, the reverse pattern.
Evaluation confidence correlated most strongly with confidence in the tool and in oneself, not with which verification strategy (running the code, tests, peer review) was actually used.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source arxiv.org