Intel / article

BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems

Source: arxiv.org
Author: Yutaro Yamada, Kei Hiroshima, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa
Date: Published Aug 5, 2026

Context

BBOWP-Bench tests whether a can decide what to search when it can observe a candidate solution's score but has no formula for calculating that score. Given a written problem, the system must choose both the variables and ranges to explore and the algorithm that will explore them. Each task includes an executable environment and a human-designed reference formulation. Benchmark.

The authors report that models can choose algorithms suited to the allowed number of evaluations, but sometimes miss important variables or choose poor ranges. This complements IR2Solve, which turns explicit optimization formulations into solver input. BBOWP-Bench exposes an earlier challenge: defining the search itself. BBOWP findings, IR2Solve method.

Key takeaways · AI-distilled

BBOWP is a new problem setting: infer BOTH a search space and an optimization algorithm from a natural-language description of a black-box task where only objective values are observable.
Each instance pairs a natural-language description with an executable evaluation environment and a human-designed baseline, so search-space design and algorithm selection are scored separately.
First results: current LLMs pick suitable algorithms for the given evaluation budget, but struggle with search-space design - identifying important variables and balancing their ranges - when descriptions are vague or highly problem-specific.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Key quotes

“Formulating an optimization problem strongly affects the quality of the final solution, yet good formulations usually require substantial expertise.”

“In BBO, the search space design, a part of the problem formulation, and the selection of the optimization algorithm are crucial for problem-solving.”

“However, they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific.”

Read the source arxiv.org