BBOWP is a new problem setting: infer BOTH a search space and an optimization algorithm from a natural-language description of a black-box task where only objective values are observable.
Each benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → instance pairs a natural-language description with an executable evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → environment and a human-designed baseline, so search-space design and algorithm selection are scored separately.
First results: current LLMs pick suitable algorithms for the given evaluation budget, but struggle with search-space design - identifying important variables and balancing their ranges - when descriptions are vague or highly problem-specific.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
If you're building agents that reason about black-box optimization tasks (e.g. hyperparameter tuning or experiment design from natural language), current LLMs reliably pick decent algorithms but often mis-infer which variables matter and their valid ranges — a concrete blind spot to guard against.
Key quotes
“Formulating an optimization problem strongly affects the quality of the final solution, yet good formulations usually require substantial expertise.”
“In BBO, the search space design, a part of the problem formulation, and the selection of the optimization algorithm are crucial for problem-solving.”
“However, they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific.”
