MongoBench: Do LLMs Know What a MongoDB Query Will Do?

AI agents that work with databases now write queries and run them on live data. Tests of such agents usually check whether the query is valid or whether the final answer is right. MongoBench, a preprint by Qibai Chen and three co-authors, tests a step in between: before a query runs, can the model say exactly which records it will match? That prediction matters most when the query deletes data, since a model that misjudges its own filter can remove records nobody intended to touch. MongoDB is a document database that selects records with filters, and several of its matching rules differ from the obvious reading. A filter for values greater than null matches nothing at all, for example, and a condition on a field also matches records where that field holds a list containing the value.

Each of the 300 items gives a model a small collection of records, about eight on average, and a filter that combines three or four of these rules. The correct answer comes from running the filter on a real MongoDB server, and a model scores only if it lists exactly the matching records. OpenAI's gpt-5.5 got 92.1% right, Anthropic's claude-opus-4-8 85.1%, DeepSeek's deepseek-reasoner 72.3% and Google's gemini-3.1-pro 70.1%. Reasoning effort was not matched across vendors, so the authors caution that the ranking mixes capability with effort. Weaker models usually misjudged only one or two records per item, which in a real delete is exactly the silent error. Difficulty tracked the kind of rule rather than the size of the collection or the number of rules combined: comparisons against null were hardest, with gemini-3.1-pro at 43% and deepseek-reasoner at 44% on those items against 96% for gpt-5.5. Mistakes were not reliably cautious either. gpt-5.5 and gemini-3.1-pro mostly predicted too many matches, while claude-opus-4-8 mostly predicted too few, the direction that would understate what a delete removes. Writing a filter did not mean understanding it: the three models that wrote the items each got 10% to 31% of their own items wrong.

The authors then tested fixes. Prepending a verified sheet with one line per rule lifted gemini-3.1-pro to 95.3% and deepseek-reasoner to 93.3% but did not improve gpt-5.5, and because the sheet named exactly the rules the items were built from, the authors treat it as an upper bound. Giving models a tool that runs the filter on the database brought gpt-5.5 and claude-opus-4-8 to 100% and deepseek-reasoner to 99.7%. gemini-3.1-pro reached 92.1%, and on 28% of items, averaged over three runs, none of the filters it actually executed passed the authors' check against the intended query. In one example it put quotation marks inside a key that was meant to select the first element of a list, so the database searched for a field with that literal name and returned nothing. The database answered correctly, but to a different query.

The authors' guidance follows from that failure: to preview a query, execute the original filter against the database and pass its result back unchanged, rather than letting the model restate or rewrite it. The benchmark covers only MongoDB find() filters, not updates or aggregation pipelines, and its items were written by the same frontier models it evaluates, with no human-written control. The paper is on arXiv. Control layers for agents, like the one Speakeasy's Sagar Batchu proposed after a coding agent deleted a production database, check whether an action is permitted. MongoBench measures a separate gap: whether the agent knows what a permitted query will actually do.