WikiBench: measuring whether a generated wiki helps coding agents

LangChain built WikiBench to check whether the codebase documentation written by its open source agent OpenWiki actually helps another agent understand a repository. Its first answer, in the WikiBench article, is that the wiki works as an index alongside the code rather than as a replacement for it. A reader agent answered the same questions about a repository three ways: with only the wiki, with only the source code, or with both. Having both produced the highest mean score at a lower cost than reading the source alone, while the wiki on its own did much worse. LangChain's explanation is that the wiki gives the reader a starting point instead of making it reconstruct the repository from scratch.

The benchmark runs on Harbor, a framework for evaluating agents on long-running tasks. Each task checks out a repository at a fixed commit, has OpenWiki generate a wiki for it, and scores that wiki with questions about the repository. The questions are generated automatically. Coverage questions follow one template, asking where an area of the code lives, what it interacts with and how to check that a change did not break anything; retrieval questions target a specific behavior. Each question comes with a generated list of facts a good answer should contain. One LLM judge checks whether each fact appears in the answer, and a second checks that it is grounded in pages the agent actually read. An answer scores the share of expected facts it got right, so three out of five scores 0.6, and the wiki's score is the average across all answers. The result therefore measures help with judged questions about a codebase, not with completed code changes.

Because the same reader can be handed the raw source instead, WikiBench measures whether a wiki helps at all, not only whether one OpenWiki version beats another. LangChain also used it to compare the models generating the wiki. Costs varied widely between runs, with the GLM 5.2 run costing $9.18 and the Luna run $0.44, and DeepSeek Flash and GLM 5.2 both performed very well, with DeepSeek Flash at roughly one-sixth of GLM 5.2's cost. The more expensive models spent their extra effort on reading: DeepSeek and GLM read about three times as many files as Luna but wrote only 1.2 to 1.5 times as many pages.