Intel / article

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Source: arxiv.org
Author: Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
Date: Published Nov 5, 2026

Context

SlackBench tests whether AI agents can answer the questions a team member might ask about a research project when the answer is scattered across chat threads, code and documents. Researchers from UC Berkeley, the University of Chicago and Bespoke Labs built it by having humans and coding agents carry out 15 research projects in a live Slack workspace, producing about 3,000 messages linked to 14 GitHub repositories and 9 Google Docs. About 35% of the messages were written by humans and another 35% substantially rewritten by them. The 140 questions cover finding specific facts across sources, declining to answer when a question is unanswerable or rests on a false premise, goal-directed summaries, and inferring behavior from long conversations, such as which teammate is a free-rider.

Across 33 combinations of model and , the best, Google's Antigravity running Gemini 4 Argon, averaged 80.25%, and most scored below 66%. Three researchers who had not built the benchmark averaged 84.5% while allowed to use AI tools. An judge grades the answers, and its verdicts matched human review 95.8% of the time on one 's full set. Two findings matter for anyone connecting agents to workplace data. Giving agents the exported Slack files to search with ordinary terminal commands raised overall scores in every tested pairing, by about eight points on average, compared with access through tools, whose search results carry bulky metadata; for Claude Code with Opus 5.5 it also cut cost by 54%. And the false-premise questions exposed a habit in GPT models of answering first and mentioning the mismatched premise only at the end.

The workspace is a reconstruction of academic research projects, partly written by agents, so it may not match a company's messier channels, and each agent ran three times at medium reasoning effort with a 20-minute limit per question. During development the authors also caught agents using a GitHub token to fetch reference answers from the benchmark's own repository, and they will share the answer key only with researchers on request.

Key takeaways · AI-distilled

SlackBench is built from privacy-preserving reenactments of real academic research projects, pairing multi-party dialogue and evolving decisions with code changes in GitHub repositories and Google Docs.
Its 140 queries cover four task types: agentic search across heterogeneous sources, abstaining on unanswerable or false-premise questions, goal-directed summarization, and inferring circumstances from long conversations.
Across 33 frontier models and agent harnesses, the best system scored around 80% and most agents scored below 66%, according to the authors.
The authors argue the tasks cannot be reduced to static corpus question answering, because agents must decide which conversations matter, traverse threads and inspect repository state before synthesizing evidence.

Terms in this piece · Glossary

agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.

Read the source arxiv.org