Artificial Analysis Endpoint Accuracy Index
- Source
- x.com
- Date
Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves. We are initiating coverage with GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 coming soon Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels and tune their inference stacks, and sometimes they simply ship bugs. We are bringing the rigor of our Artificial Analysis Intelligence Index to measuring endpoints, so developers can pick providers on accuracy, not just price and speed We benchmark each serverless endpoint against our own self-hosted reference deployment of the official weights, where 100% represents matching the reference. An endpoint is at reference parity when its result falls within the 95% confidence interval of the reference. Coverage is live for GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 accuracy coverage launching soon Key elements of the Endpoint Accuracy Index: ➤ Three areas, equally weighted: tool calling (BFCL-500, 500 questions, 3 repeats), scientific reasoning (HLE-250, 250 questions, 10 repeats) and long context recall…

Context
An open-weights model can be downloaded and run by anyone, so many companies sell API access to the same model, and the Endpoint Accuracy Index from Artificial Analysis tests whether those services give the same quality of answers. The company's post says providers trade accuracy for speed and cost: they quantize weights (store the model's numbers at lower precision so it runs cheaper), write custom kernels, tune their serving software, and sometimes ship bugs. Artificial Analysis says the index lets developers choose providers on accuracy, not just price and speed.
Each endpoint is scored as a percentage of a reference deployment that Artificial Analysis runs itself from the official weights, at the precision and settings the model's developer recommends, so 100% means matching that reference, not answering every question correctly. The score blends three equally weighted test sets: 500 tool-calling questions run three times, 250 hard scientific reasoning questions and 25 long-document recall questions, each run ten times. An endpoint counts as at parity when its result falls within the 95% confidence interval of the reference, a test that allows for run-to-run variation in both. Every endpoint is run at its highest supported reasoning mode, output length and , so the index measures the service a customer actually gets, limits included. The methodology page adds that results are dated snapshots, since providers can change their setup.
The first results, published with the August 4, 2026 announcement, show large differences for some models. On GLM-5.2, the post says endpoints with the tightest output limits cut answers off before the model finishes reasoning and score half the reference or less on the reasoning set; the lowest overall score in its chart is 52%. On gpt-oss-120b, providers handle tool calls differently, and some score 22% on the tool-calling set against 37% for the reference. DeepSeek V4 Pro is the exception: the post says most endpoints sit at parity, and DeepSeek's own endpoint scores slightly above the reference. Artificial Analysis's GLM-5.2 provider comparison compares hosts on speed and price; this index adds a check on whether a cheaper or faster host is giving up accuracy to get there.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
Checking sign-in…
Loading comments…






