Vibeleaderboard
← All Intel
Intel / post

Announcing the Artificial Analysis Search Index, benchmarking how search API pro

Source
x.com
Date
Artificial Analysis@ArtificialAnlys

Announcing the Artificial Analysis Search Index, benchmarking how search API providers perform on quality, cost, and speed when used by an agent. We are initiating coverage with Parallel, Exa, Firecrawl, You (dot) com, Tavily, Keenable, and Brave Search is one of the most important tools for agents. Search providers make different choices about how they search, rank, and package results, and those choices change what the model reads and how it acts. We are expanding our benchmarking coverage to search APIs, so developers can pick a search provider on measured quality, cost, and speed. Each provider result pairs a search API provider with the same model, GPT-5.6 Luna (medium). The model runs inside Stirrup, our open-source agent harness, with tools for searching and fetching pages from the web - only the search provider behind the search tool changes. At launch, the leaderboard covers 11 results across 7 search providers, and we’ll keep expanding coverage as we look to provide the most accurate and comprehensive benchmarking of search providers for AI agent usage. Key elements of the Artificial Analysis Search Index: ➤ Three equally weighted benchmarks: the Search Index is the…

Read the full post on X

Context

When an researches a question on the web, the search service behind its search tool decides which pages and snippets the model gets to read. Artificial Analysis, which benchmarks AI models and services, launched a Search Index that isolates that choice. Every result uses the same model, GPT-5.6 Luna at medium reasoning effort, inside the same open-source , Stirrup, with up to 25 turns per task and at most 10 results per search. Only the provider behind the search tool changes. Coverage at launch was seven providers, among them Parallel, Exa, Firecrawl, Tavily and Brave.

The score averages three benchmarks with equal weight: DeepSearchQA, 900 broad research questions that need many searches; a 200-question hard subset of BrowseComp, facts that need several hops of browsing; and a 600-question private subset of Artificial Analysis's AA-Omniscience. For a baseline, the same model also answers each question once with no tools. That baseline scored 33. With search, results ranged from 65 to 75, so any of the tested providers roughly doubled what the model could answer from its own knowledge. Parallel, Exa and Firecrawl led with 75, 74 and 73, a gap small enough that cost and speed may matter as much as rank when choosing among them.

The more useful finding is about cost. The index adds what the task spends on search to what it spends on model , and that changes which option is cheaper. Parallel's advanced tier spends more on search per task than its basic tier, but Artificial Analysis reports that the better results cut the model's use by over 40%, bringing the total per task to $0.084 against $0.11 for basic, with a higher score. Speed behaved the same way. Parallel's turbo tier answered searches in 0.51 seconds on average against 1.03 for basic, yet scored 67 to basic's 73, and the two finished tasks in about the same total time. A developer comparing search APIs by their price list or per-query latency could pick the wrong one. These figures come from one model and harness setup, and the post does not say how they would shift with a different model.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
More from Artificial Analysis
Recommended reads
Comments

Checking sign-in…

Loading comments…