Intel / article

AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

Source: arxiv.org
Author: Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui
Date: Published Nov 11, 2026

Context

AI models that write ordinary code well mostly fail at AgentScript, the language Salesforce's Agentforce platform uses to build conversational agents, according to a benchmark from Salesforce researchers and William & Mary. AgentScript is a domain-specific language, a small purpose-built language in which a developer declares what an may discuss, which actions it may call and which apply, and the platform then runs the agent. The paper notes the language is actively developed and absent from public code, so models have had little chance to learn it. Its benchmark, AgentScript-Eval, has 100 tasks drawn from real agent-development work: 30 ask for an agent written from a specification, 40 ask for a broken one to be repaired, and 30 ask for a working one to be improved. A program that does not compile scores zero, and one that compiles is run through scripted conversations that check its behavior.

Asked directly with no examples, five models from Anthropic, OpenAI and Google did poorly. The best, Claude Opus 4.8, solved 25.0% of tasks under the paper's pass@3 measure, in which one generated program is run three times and counts if any run passes every check, and no model solved any of the tasks that start from scratch. Gemma-4 31B, the only model, compiled the most programs, 48.0%, and on partial credit, the average share of checks passed, it came close to Opus (28.1% against 31.3%) while using the fewest tokens, although it fully solved fewer tasks.

The setup around the model moved results more than the choice of model. Given a coding agent that could compile and run its own program and try again, Opus's partial-credit score rose from 31.3% to 46.5%; the abstract's description of this as roughly doubling fits Claude Sonnet 5's rise from 17.4% to 40.4% better than Opus's. How the AgentScript guidance was written also mattered. The same pack, a set of reference instructions for the agent, solved 28% of tasks as plain documentation and 14% as a live plugin, though that switch changed the catalog of guidance as well as its delivery. Reorganizing the plugin's authoring instructions so their entry file shrank from 830 lines to 461, with the detail moved into smaller reference files, raised it to 37%, the best result. These figures are Opus runs scored with a single execution (pass@1), so they are not directly comparable with the 25.0% above.

Even the best configuration solved just over a third of tasks. Compile rates reached 98% there, but compiling did not mean behaving correctly: 40 of the 100 tasks still earned only partial credit. The results come from 100 tasks and single runs, the agent experiments used only two models, and the benchmark was built by the company that makes the language.

Key takeaways · AI-distilled

AgentScript-Eval holds 100 execution-graded tasks in AgentScript, the declarative DSL Salesforce uses to build Agentforce agents, split across authoring from a spec, debugging a broken definition, and optimizing a working one.
Zero-shot, the best of five models reached only 25.0% pass@3, and every model scored 0% on authoring an agent from scratch.
Scale and price predicted success poorly: a 31B open-weight model had the highest compilation rate tested (48.0%) and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.
Adding an agentic compile-and-edit loop was the largest single improvement, roughly doubling partial-credit accuracy.
Guidance wording mattered more than packaging: one skill pack scored 28% as plain documentation and 14% as a live plugin, and rewriting a single skill in that plugin raised it to 37%, the paper's best result.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.

Read the source arxiv.org