What Stops a Small Language Model From Driving a Database Agent
Source
Cevheri Bozoglan, Yusuf Gundogdu, Abdullah Kaya, Koray Sirin
Author
Cevheri Bozoglan, Yusuf Gundogdu, Abdullah Kaya, Koray Sirin
Date
Key takeaways · AI-distilled
The study drove 39 open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models plus one hosted control against a production SQL-client AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → over 11 days: 8,199 runs, 110,711 ledger events, 14,008 refused tool calls.
Of 2,100 model-attributed losses, 75.7% came from runs that had invoked at least one tool, and that majority held in 99.7% of clustered resamples across models, not just runs.
Within failures, transport (tool used but no deliverable came through) was the largest class at 36.2%; capability (no tool invoked at all) was smallest at 17.3% - though the authors flag this ordering as corpus-specific, holding in only 74.5% of resamples when clustered by model.
Production ledgers logged refusal codes but never the model's actual arguments, hiding five server defects for ten days - one required a field on one tool while forbidding it on the sibling tool that composed it. Fixing all five moved six models by 6-21 cells out of 30.
A benchmarking confound: with no context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → cap, one 7.1GB model was admitted at its full 262,144-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → window and consumed 51GB on a 64GB machine, producing runs indistinguishable from a timeout in ordinary logs.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Pinpoints that small local-model agent failures are dominated by fixable transport/argument-shape issues rather than reasoning capacity, directing engineering effort toward tool-call plumbing rather than swapping in bigger models.
articleEngineering Reliable Coding Agents: Evaluating and Operating the System Around the ModelStephanie Jarmak
articleToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling AgentsYiShan Zheng, Yuan Wu, Yi Chang
articleAn Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming TasksChandimal Adikari, Nandika Herath