Introducing Research Eval A Benchmark For Search Augmented Llms
Source
reka.ai
Author
Reka AI
Date
Context
Reka AI designed Research-evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → to test whether a web-searching model assembles the information a request actually requires. The benchmark announcement describes 374 questions, each paired with a checklist used by a language-model judge. A request for several fees and their total, for example, requires every amount and the arithmetic to be correct. Reka's in-house annotators, whom it calls AI tutors, refined the requirements and removed ambiguous or inconsistently judged examples. The resulting test evaluates completeness against explicit requirements across varied web sources, making its scoring criteria more inspectable than a general impression that an answer sounds useful.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleDeepsecBenchMalte Ubl
articleSearchAuditor: Auditing and Attributing Failures in Long-Horizon Search AgentsZhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
articleBBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word ProblemsYutaro Yamada, Kei Hiroshima, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa