Intel / article

Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

Source: arxiv.org
Author: Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul
Date: Published Oct 12, 2026

Context

A prompt optimizer needs feedback that explains what went wrong. In a study of a research agent, researchers tested an optimizer that rewrites the instructions used to produce cited reports. Feedback distinguished problems such as irrelevant sources and sources that failed to support a claim.

In the verifier comparison, collapsing that diagnostic feedback into one aggregate score left optimized prompts 0.021 below the unoptimized baseline on held-out topics on average, while the reference setup with separate feedback gained 0.075. The comparison used two optimization seeds per configuration and 120 held-out tasks. The authors treat the cell means as directional, so it supports diagnostic feedback for this and citation-scoring setup, while leaving broader generalization unestablished.

Key takeaways · AI-distilled

The held-out citation-reward gain of 0.079 has a bootstrap 95% CI of 0.051 to 0.107 and replicated across three seeds, under a split where no topic is shared between train, validation and test.
Nearly half the winning prompt was a memorized answer key, yet an ablation showed that content earned no reward even on the exact training questions. The authors attribute the validation-to-held-out gap to small-sample selection under a noisy judge.
Decomposed per-dimension feedback was the causal ingredient: collapsing the verifier to a single scalar reward made held-out transfer negative.
The gain held under an out-of-family judge and was largely reproduced by a hand-written version of the discovered recipe. The authors call for held-out topics, multiple seeds and verifier ablations when evaluating prompt optimizers.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Read the source arxiv.org