How is AI really affecting your job? @Cohere_Labs released the Agentic Task…

Cohere's research arm, Cohere Labs, built the Agentic Task Ecosystem, a dataset of over 690,000 tools meant to represent tasks people actually do in their jobs, then tested whether an AI agent could complete each task independently using the relevant tool. Only 2.6% of those attempts passed.

That low pass rate applies specifically to unaided, independent completion of a modeled real-world task with an available tool, not to a model's ability to assist a person with parts of that task, which is a narrower and different claim than a single benchmark score would suggest.