Vals AI launches independent Web Search Index benchmark
- Source
- x.com
- Date
We realized that web search wasn’t being measured correctly and we weren’t alone. Other companies have created their own benchmarks to tackle this problem, but kept running into the same issue: data contamination, realism, and answer leakage. When benchmarks measure the wrong thing, customers can't tell which search tool is best. We believe the teams building these tools deserve to have a third-party benchmark that measures the work their users actually need done. We’re excited to share that @ExaAILabs, @KeenableAI, @p0, and @tavilyai are joining us as partners on the Vals Web Search Index: one benchmark, the same settings for everyone, run independently. Check out our latest blog to learn more: 🧵
Context
The Web Search Index asks whether changing an 's search tool changes its final answer. Vals AI's announcement proposes one , with the same settings for every search provider, run independently, and names Exa, Keenable, Parallel and Tavily as partners. Its method diagram is labeled 'Same agent. Swap the search tool.': the model and the , the software that coordinates the agent, are marked as the same, the search tool is swapped, and the final answer is graded, with finance and legal shown as the answer types. Holding those other components constant helps isolate the search tool's contribution to completing the work. The announcement establishes the comparison design, without declaring a winning provider.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Checking sign-in…
Loading comments…