Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
Source
developer.nvidia.com
Author
Michelle Horton
Date
Context
NVIDIA's SkillEvaluator tests whether packaged AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → instructions help by running matched tasks with and without the agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →. Prompt, model, inputs and grading criteria stay fixed within each agent setup. In NVIDIA's reported benchmark of more than 300 of its own verified skills, run on two agent harnesses, average correctness scores rose from 46 to 87.
That is a 41-point change in a graded score, rather than a task pass probability. Most skills received one attempt per task, and the report provides no confidence intervals. The useful contribution is a repeatable comparison for testing a skill's effect on specified tasks.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleHow to Evaluate AI Agents From Tool Calls to Task CompletionElizabeth Goodman
articleNVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per WattElizabeth Goodman
articleOptimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo GymTanya Lenz