Is it agentic enough? Benchmarking open models on your own tooling
- Source
- huggingface.co
- Date
Context
Instructions intended to help a coding can also confuse it. In a Hugging Face study, researchers tested agents using the Transformers machine-learning library with different combinations of installed software, source code and packaged instructions. They measured effort and failures as well as correct answers, on tasks with exact-match answers.
The packaged instructions reduced time and turns for the largest open models, but Qwen3-14B mistook a documented command-line program for a tool it could call directly. In 39 of its 56 runs with the instructions it either issued a call to a tool that had never been registered or concluded it could not run a model. Its answer-match rate fell from 67% with the basic installation to 43% with the instructions. The authors’ reading of the traces is that a change that reduces work for strong models can add ambiguity for smaller ones.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
Checking sign-in…
Loading comments…