Vibeleaderboard
← All Intel
Intel / post

Parallel (turbo and advanced) and Firecrawl make up the Pareto frontier for Arti

Source
x.com
Date
Artificial Analysis@ArtificialAnlys
Thread · 2 parts

Announcing the Artificial Analysis Search Index, benchmarking how search API providers perform on quality, cost, and speed when used by an agent. We are initiating coverage with Parallel, Exa, Firecrawl, You (dot) com, Tavily, Keenable, and Brave Search is one of the most important tools for agents. Search providers make different choices about how they search, rank, and package results, and those choices change what the model reads and how it acts. We are expanding our benchmarking coverage to search APIs, so developers can pick a search provider on measured quality, cost, and speed. Each provider result pairs a search API provider with the same model, GPT-5.6 Luna (medium). The model runs inside Stirrup, our open-source agent harness, with tools for searching and fetching pages from the web - only the search provider behind the search tool changes. At launch, the leaderboard covers 11 results across 7 search providers, and we’ll keep expanding coverage as we look to provide the most accurate and comprehensive benchmarking of search providers for AI agent usage. Key elements of the Artificial Analysis Search Index: ➤ Three equally weighted benchmarks: the Search Index is the average of DeepSearchQA (900 broad research questions that need many searches, graded with an F1 score over answer items), BrowseComp (a 200-sample hard subset of facts that need multi-hop browsing), and AA-Omniscience (a 600 question private subset, balanced across 6 domains) ➤ Same agent, different search provider: the agent has 25 turns available to complete each task. Its web search tool returns the search provider's native response payload (with content modes standardized to snippets), with a maximum of 10 results and contamination sources filtered out ➤ Model-only baseline: we compare search agent results to the same model answering single-shot without tools, showing how much each provider lifts the model above its internal knowledge ➤ Cost and Time per Task: we aggregate the time and cost spent on both model inference and search. This is key - search APIs have different cost and latency structures, but these can be offset where they help an agent use fewer turns and save on costly language model inference Key results: ➤ Parallel, Exa, and Firecrawl have the strongest overall performance, with Artificial Analysis Search Index scores of 75, 74, and 73 respectively at launch ➤ All search providers tested substantially improve knowledge-based benchmark performance: the model only baseline scores 33 on the Search Index, while search-included provider results score between 65 and 75 ➤ Focused search results reduce spend on model inference: Parallel Search (advanced) search costs more per task than Parallel Search (basic) but less per task in total ($0.084 vs $0.11). Higher quality results cut the model's token use by over 40% in this case, more than offsetting increased search costs while reaching higher benchmark scores ➤ Fast search calls do not guarantee fast tasks: Parallel Search (turbo) has the fastest average search calls among Parallel's tiers (0.51s per query vs 1.03s for Parallel Search (basic)) but the basic tier scores higher on quality (73 vs 67) and the two land close on total time per task

Parallel (turbo and advanced) and Firecrawl make up the Pareto frontier for Artificial Analysis Search Index vs. Cost per Task, and represent the best performance for the blended cost available today. Total cost per task combines both search API and model cost, which means we see variable pathways to low cost task completion. Firecrawl Search scores 73 on the AA-Search Index at $0.075 total cost per task which is only ~40% search cost. However, it has the slowest time per task on the board (55.0s). Parallel (advanced) on the other hand cost 12% more per task to reach an Index of 75, while reducing cost spent on model inference due to its search efficiency.

Context

In Artificial Analysis's new Search Index, the search option with the smaller search bill was not always the cheaper way to finish a task. The index tests search APIs, the services an calls to search the web, by giving the same model, GPT-5.6 Luna (medium), the same open-source and 25 turns per task, and changing only the search provider behind its search tool. The score averages three research benchmarks: DeepSearchQA, a hard subset of BrowseComp and a private subset of AA-Omniscience. At launch it covered 11 results from 7 providers. The model answering alone, without tools, scored 33; with search, results ranged from 65 to 75, led by Parallel at 75, Exa at 74 and Firecrawl at 73 (Artificial Analysis on X).

Cost per task counts both the search fees and the model's own , and the two pull against each other. Parallel's advanced tier charged more for search than its basic tier but cost less per task overall, $0.084 against $0.11, because better results cut the model's use by more than 40%. Speed worked the same way: Parallel's turbo tier had the fastest searches of Parallel's tiers, 0.51 seconds per query against 1.03 for basic, but scored 67 to basic's 73 and took about as long per task. Firecrawl scored 73 at $0.075 a task, only about 40% of it search cost, but was the slowest on the board at 55.0 seconds per task. Parallel advanced reached 75 for about 12% more (cost breakdown).

Those three results, Parallel turbo, Parallel advanced and Firecrawl, form what Artificial Analysis calls the Pareto frontier for score against cost: for each, no other tested result is both cheaper and higher scoring. The rankings describe one model in one harness at launch, as measured by Artificial Analysis. The index kept adding providers afterward, as in later coverage of Perplexity's Search API.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
More from Artificial Analysis
Recommended reads
Comments

Checking sign-in…

Loading comments…