Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks
- Source
- arxiv.org
- Author
- Timothy Kassis
- Date
Telling an agent it is a senior chemist or a veteran statistician is one of the most common habits in prompt writing. This paper puts the habit to a direct test. The authors built 503 profession-specific agent profiles and ran them against minimal prompts and control prompts across nine science benchmarks. Profiles matched to the task did not deliver a clear accuracy improvement. They did change the bill: matched profiles produced up to 2.3 times more output tokens, and per-call cost rose by a factor of 2.2 to 4.5. The practical reading is narrow but useful. A persona block is not free decoration, because it shapes how much the model writes, and on these benchmarks the extra writing bought nothing measurable. Teams carrying long role descriptions in production system prompts can run the same comparison on their own evaluation set: a minimal prompt against the persona version, scoring both accuracy and tokens. If the persona does not move accuracy, removing it is a cost cut with no quality tradeoff. The results cover science benchmarks, so other domains still need their own check.

Context
Detailed professional personas did not consistently improve scientific-answer accuracy in a study using Google's Gemini 3.8 Flash. The reported experiment compared profession-specific instructions with a minimal assistant prompt and other controls across nine text-based science benchmarks.
Among 4,488 questions completed under all five conditions, the average accuracy difference from the minimal baseline was minus 0.6 percentage points, with uncertainty spanning zero. Estimated cost per successful call was 2.2 to 4.5 times higher with the professional profiles. For this model and these tasks, adding detailed role instructions increased expense without a demonstrated accuracy benefit.
- Profiles were tested against four controls: a minimal helpful-assistant prompt, the profile's opening role sentence, a generic scientific-rigor guide, and a profile from an unrelated field, using Gemini 3.8 Flash in the Pi .
- Across 4,488 completed items on nine benchmarks, the average accuracy difference between profile and baseline was -0.6 points (95% interval -1.5 to +0.2), with no showing a clear gain.
- On 60 tool-using BioMysteryBench problems, profiles lowered the mean solve rate from 56.7% to 46.7%, largely because profile runs hit and time limits more often.
- Longer prompts did help on SuperGPQA when provider API drops struck (71.6% vs 54.0% correct on first pass), but generic and mismatched prompts did as well, pointing to length or formatting rather than domain expertise.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleScientific Agent Skills: A Library of Procedural Knowledge for Research AgentsTimothy Kassis, Vinayak Agarwal, Yuhuan He, Darshil Patel, Aubrey M. Brueckner
articleAutomating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski
articleReflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level OverfittingElias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul
Checking sign-in…
Loading comments…