The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →'s turn cap partly measures who finishes in time. With the cap relaxed, open-weight models matched or beat closed ones at under a tenth of the cost, which changes model choice for security agents.
Key takeaways · AI-distilled
On CTI-REALM, a run that hits the message limit scores zero on every checkpoint, including ones already passed. At the published limit the leading open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → and closed models cannot be distinguished.
The authors add stage-level metrics that execute each AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s detection queries against the telemetry store. Different models lead different stages, so they suggest routing each workflow stage to its best model.
Among leading models, price was a poor guide: models with similar rewards differed in cost by more than an order of magnitude, and the share of output spent on thinking did not track reward. The study covers 19 open-weight and 7 closed models.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleSelecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 ModelsParishruthi Ganesh, Gerry Dozier, Cheryl Seals
articleWhy Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial MarketsJillian Ross, Eric So, Zoe De Simone, Charles Pozniak, Andrew W. Lo
articleTwo Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and MissesJannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu