Vibeleaderboard
← All Intel
Intel / post

OpenAI web search lands mid-pack on the Artificial Analysis Search Index

Source
x.com
Date
ArtificialAnlys@ArtificialAnlys

OpenAI Web Search debuts on the Artificial Analysis Search Index at 74, the 5th best provider behind Perplexity, Octen, Parallel and Brave @OpenAI Web Search is the first integrated first-party search tool on our board. Instead of an agent calling a Search API in our Stirrup harness, GPT-5.6 Luna (medium reasoning) calls OpenAI's built-in web_search tool, and OpenAI runs the whole search loop inside one Responses API call. All setups use the same underlying model, GPT-5.6 Luna (medium reasoning). At ~$0.05 per task, OpenAI Web Search costs less than Parallel (advanced, $0.06) and Perplexity (medium, $0.07), but about twice as much as Octen ($0.024), which scores 3 points higher. Key benchmarking results for OpenAI Web Search (medium search context): ➤ 5th best provider on the Artificial Analysis Search Index: OpenAI Web Search scores 74, a 41-point lift over the same model with no search (33). It places 7th of 26 products, level with You . com (highlights), Nimble (standard) and Exa (auto) at 74, behind the three Perplexity variants (77 to 80), Octen (77), Parallel (advanced) and Brave (LLM context) at 75 ➤ OpenAI Web Search performs best on AA-Omniscience, our proprietary…

Read the full post on X

Context

OpenAI's built-in web search is good at looking up a single fact and weaker at research that chains several searches together, and it lands in the middle of Artificial Analysis's Search Index. The index, launched in August, measures how well an answers questions when it is given a particular search product. Normally the agent calls an outside search API from inside Artificial Analysis's own , Stirrup. OpenAI's tool works differently: the model calls OpenAI's web_search tool and OpenAI runs the whole search loop itself inside one API request. In its October 6 post, Artificial Analysis says it tested this with the same model it uses for every entry, GPT-5.6 Luna at medium reasoning.

With search the model scores 74 on the index, against 33 with no search at all. That puts OpenAI 5th among providers and 7th of 26 products, level with entries from You.com, Nimble and Exa, and behind three Perplexity variants (77 to 80), Octen (77), Parallel and Brave (75). The split across benchmarks shows where it gains and loses ground. On AA-Omniscience, Artificial Analysis's own factual-answer test, it reaches 72 percent, 3rd of 26 and one point behind the leader, Firecrawl. On BrowseComp, where an answer requires several searches and reasoning across their results, it scores 73.5 percent and ranks 13th, while the Perplexity variants and Octen score 85 to 87 percent.

Cost is where the built-in tool looks better. OpenAI charges $10 per thousand web_search calls and bills the retrieved pages as model input , and Artificial Analysis reports about 40,000 billed input tokens per task, search results included, against 125,000 for the leanest outside API, Perplexity at its low setting. At about $0.05 per task it is cheaper than 17 of the 25 search APIs on the board, whose median is $0.067. The exception matters for anyone choosing: Octen costs about $0.024 per task and scores 3 points higher. Because OpenAI's own loop replaces the Stirrup harness for this entry, the gaps compare OpenAI's whole search setup with each API run inside that harness, all using Artificial Analysis's one model and settings.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
More from ArtificialAnlys
Recommended reads
Comments

Checking sign-in…

Loading comments…