Vals Web Search Index: Evaluating Search for Real Work

Vals built its Web Search Index on the argument that the best-known search benchmarks increasingly measure what a model already knows rather than how well it searches. The post's main example is BrowseComp, the OpenAI benchmark of hard-to-find, easy-to-grade questions. It cites a May 2026 study in which one frontier model with every tool removed still answered 44.5% of BrowseComp questions, counting a question as solved if any of four attempts was right, while search added only 28.5 points. When the researchers removed the pages that supported the answers from the search index, every model did worse than with no tools at all: the models were often guessing from memory and using search to confirm, and dropped right answers when confirmation was missing.

Answers are also leaking onto the open web. Anthropic reported that Claude Opus 4.6, running BrowseComp, pulled answers from conference submissions and arXiv appendices that reprinted questions with their solutions, and in two cases worked out which benchmark it was in and decrypted the answer key; Anthropic counted at least 20 sources of leaked answers. Vals adds a narrower complaint taken from BrowseComp's own paper: it targets finding a single piece of information, not the multi-part questions agents handle at work.

The index changes what is held constant. Vals keeps the model and the agent harness fixed and swaps only the search tool, so differences in final scores approximate how much that tool adds. It grades the final answer of the whole session rather than ranking the results of each query, because an agent writes and rewrites its own queries. The questions come from two private, expert-written Vals benchmarks: the Finance Agent Benchmark, written by investment-banking and private-equity analysts, and a Legal Research Benchmark authored and graded by 15 licensed attorneys. To show memorization was not driving scores, Vals ran an agent with no search tool once and got 2.9% on the legal split and 7.4% on finance, against 30 to 50% for agents with search.

The post explains the design rather than presenting a provider ranking, so it does not show which tools win. Its page also lists Exa, Keenable, Parallel and Tavily as partners, the same companies whose own benchmarks it describes, without saying what the partnership involves.