DeepInstructor proposes evaluating research ideas by retrieving evidence from past peer reviews. It organizes material from 58,607 reviews into an Experience Graph, then uses an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → to retrieve evidence relevant to the aspect being assessed, such as novelty or feasibility. This gives the evaluation a traceable basis in recorded scholarly judgments. In the paper's abstract, the authors report better agreement with human judgments on a dataset of controlled comparisons between ideas. That measures how well the evaluator matches those judgments; it does not establish that its preferred ideas produce better scientific results.
Key takeaways · AI-distilled
The authors argue LLMs now generate research ideas faster than they can be judged, and that existing evaluators rely on parametric model knowledge or unstructured retrieval rather than experience-groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → reasoning.
They release DeepInstruct, a dataset of controlled pairwise idea comparisons along three dimensions: novelty, significance and feasibility.
DeepInstructor improved Hit@1 and Hit@2 alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → with human judgments by 24.4% and 29.7% over existing baselines, with the retrieved evidence making each evaluation traceable.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
articleAgentic Data Engineering for LLMs: System Design and a Controlled Empirical StudyRuomeng Ding, Qianli Shen, Zhaoyang Han, Meixin Chen, Daoyuan Chen, Yaliang Li
articleLabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI AgentsLei Liu, Yikun Zhang, Jialin Chen, Wanjia Zhao, Rex Ying, Wengong Jin, Hua Xu, James Zou, Tianyu Liu, Hongyu Zhao
articleXAI-Arena: Can LLMs Assess the Quality of XAI Explanations?Yanfei Hu Fleischhauer, Alona Zharova, Nadja Klein, Stefan Feuerriegel