Verifier-driven tuning took a chip design agent from 55% to 95%

A chip-design agent from Siemens EDA, the chip-design software unit of Siemens, became far more reliable without a change of model. The team kept the same model and harness, Anthropic's Claude Opus 4.8 running in Claude Code, and revised the instructions and tool descriptions the agent works from. The mean score on its benchmark rose from about 55% to about 95%, while agent turns fell from 155 to 53 and output tokens from about 81,000 to about 29,000, according to a developer blog post by NVIDIA and Siemens engineers. The changes were guided by a verifier that grades the agent's finished files and by NeMo Gym, NVIDIA's open library for evaluating agents, which records every attempt so engineers can see where it went wrong.

The task is standard-cell characterization: producing models of how each logic cell on a chip behaves for timing, power and noise across operating conditions. Those models are stored in Liberty (.lib) files, and they must be regenerated and validated whenever a semiconductor foundry issues a new process design kit, the package of transistor models, design rules and process, voltage and temperature definitions (PVT corners) that ties manufacturing to design tools. The agent has to set up tools, run large SPICE circuit simulations on distributed compute, generate the Liberty files, debug them and validate them. The authors say mistakes in this chain tend to produce plausible output, such as a wrong threshold or a missing corner, rather than an obvious failure, and an early setup error can carry through every later stage. Checking whether individual tool calls succeeded does not catch that.

So the verifier inspects the final Liberty files against known reference files. Four checks are pass or fail. Coverage asks whether every expected cell and corner is present, validity whether the files are well formed and usable by validation tools, relative validation whether the output adds failures beyond those already in the reference, and integrity whether the workflow actually produced the results instead of reusing, truncating or reformatting existing values. If all four pass, the score is the fraction of timing and power values within tolerance; if any fails, the score is zero. The integrity check exists to stop the agent from appearing to succeed by recycling old numbers, which the authors describe as cheating. NeMo Gym starts each attempt from a clean state and logs the agent's reasoning traces, messages and tool actions, which let the team trace failures to ambiguous instructions, unproductive actions or the wrong choice of tool.

The fixes targeted the agent's instructions and the way its tools are presented to it. The agent reaches its tools through the Model Context Protocol (MCP), a standard way of exposing tools to an AI agent, and reads each tool's description to decide what to call. The team simplified overlapping file and shell tools and gave tool inputs explicit JSON schemas, precise definitions of each argument, which made the characterization tools easier for the agent to call. The evidence for the result is narrow: about 25 rollouts of 1.5 to 2 hours each, run on two cells across 36 corners of the open-source GF180MCU process kit rather than its full 229 cells, scored by a verifier the same teams wrote. The post does not say how those runs divide between the starting and final versions or how other models perform. What the post does lay out is a method the authors say other engineering teams can reuse: grade the finished artifact, make the grader catch shortcuts, and use the recorded failures to fix the agent's instructions and tools.