Frontier models still score below 50% on Kubernetes incident root-causing, which tempers expectations for autonomous SRE agents.
Key takeaways · AI-distilled
ITBench-AA was built by Artificial Analysis with IBM, whose Software Innovation Lab developed the dataset. Its 59 SRE tasks include 19 held-out ones, with FinOps and CISO tasks planned.
Each task supplies a Kubernetes incident snapshot (alerts, events, traces, metrics, logs, topology); models run in the open-source Stirrup agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → with shell access, a 100-turn cap and 3 repeats per task.
Scoring is precision at full recall: missing any true root cause scores 0 for that repeat, and naming extra contributing entities lowers the score.
More turns did not help: GPT-5.5 (xhigh) averaged 31 turns for about 46%, Gemini 3.1 Pro Preview 83 turns for 30%. Models that over-investigate tend to flag upstream fault mechanisms or co-occurring symptoms.
Gemma 4 31B scored 37% at $0.14 per task, and GLM-5.1 led open-weights models at 40% for $1.23 per task, matching Gemini 3.5 Flash (high) on score at lower cost.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
