Company docs didn't help our AI agent. Four shortnotes doubled accuracy
- Source
- contextflo.com
- Author
- depthfirst
- Date

Context
An answering questions about a company's data got no more accurate when given the company's documentation, but four short notes about how the business works took it from 36% to 76% of tries correct, in a test run by Contextflo, whose product connects AI agents to company databases (Contextflo). The agent ran Anthropic's Claude Haiku 5.5 on DABstep, a public from the payments company Adyen and Hugging Face that imitates a payments business with realistic, messy data: 138,236 card payments, a 1,000-row price list and 30 shops. On 42 held-out questions, tried three times each, it got 40% of tries right with only the tables, 36% with the docs added, and 76% with the docs and notes. Cost per try fell from about 5 cents to about 1.3 cents.
The docs explained what each column means, but not the rules the business uses to answer questions. The first note says a blank in a price row means "any": the manual covered one kind of blank but not the empty lists the price list also uses, so the agent treated blanks as not applying, ignored most of the price list and came up with fees that were too low. The second says one payment can match several rows of the price list and pays all of them, something the docs never state. Without it the agent found one matching row and stopped. In the exploration round, that note alone lifted accuracy from 31% to 59%, the largest jump. The docs did matter for cost, cutting it by about two thirds without changing accuracy, because the agent stopped running query after query to guess what columns meant. Spotify reached a similar conclusion building its internal data assistant: a table's schema does not say what a term such as "active user" means.
Contextflo wrote the notes from the agent's wrong answers on 25 exploration questions, then froze the notes, docs, instructions and scoring before testing on the 42 new ones. Most of the gain came on question types the exploration round had covered, now asked about new shops, months and amounts, where accuracy rose from 26% with the docs to 70% with the notes. The four new question types were easier, with 75% right from the tables alone. A note can also break things: the rule about adding up every price turned a question the agent had answered right on all three tries into zero for three, because it worked out the right answer and then changed it to follow the note. The company also corrected its first exploration-round score from 77% to 63% after finding its rebuilt answer sheet too lenient. This is one vendor's test of one model on one small dataset, but the runs, notes and questions are all published for others to check.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…
