How NVIDIA Uses AI Agents to Streamline AI Factory Operations

NVIDIA has described a team of AI agents it uses to help the engineers who keep its own GPU clusters running, and has released the framework as experimental open-source software. Its November 4 developer blog post says the AI Factory Operations Agent serves site reliability engineers working on internal DGX SuperPOD clusters. Normally, finding why a training job slowed down often means checking separate dashboards for compute, network, hardware and job scheduling.

An engineer asks a question in a chat interface, and a central runtime routes it to a specialized agent. One handles failed or degraded training jobs. Another traces hardware faults such as PCIe and ECC errors, GPU problems, failing NVMe drives and thermal anomalies down to a component. Others cover monitoring data, documentation search, Kubernetes state and cluster management. Each reaches the infrastructure through Model Context Protocol (MCP) servers, a standard way to give an AI model a fixed set of approved tools. The agents run on NVIDIA's Nemotron open models served locally, so the whole system can stay on-premises.

The design starts from read-only access. Kubernetes inspection goes through a server-side tool that runs a pinned kubectl binary with read-only credentials, and the per-session sandbox where the agents run shell commands never receives those credentials. NVIDIA's example permission check shows the agent can list pods and configmaps but cannot read secrets or create or update pods. NVIDIA lists "read-only first" among its lessons from production use: accurate investigation has to earn the engineers' trust before any automation.

The post publishes no results. NVIDIA says it measures root-cause accuracy and how often engineers agree with the agents, but gives neither figure, so its aim of reducing manual work cannot be checked from this post. It also warns that the framework's APIs, configuration and deployment requirements may change between releases.