Rippling's four-tier agent eval stack, from mocks to deploy gates

Rippling tests its AI agents in four stages that move from recorded fake responses to the real production system, according to a post by LangChain and LangChain's longer case study about the company. The agents being tested are a supervisor that routes each request to five to seven specialized sub-agents, which query data across Rippling's products, search help documents and HR policies, or make changes such as uploading bonuses and pre-filling records for new hires.

The first stage runs on every commit, locally, using pre-recorded responses and fixed test data so that nothing outside the developer's machine is called. After code is merged, 300 to 400 queries run against a full Rippling sandbox with live API calls, to check the assembled system before deployment. Around ten critical scenarios then run against real systems, and every deployment depends on them passing. After release, scheduled runs against production data several times a day watch the live system's health. Results from the stages are uploaded to LangSmith, LangChain's product for storing agent traces and evaluation results.

The case study does not explain the ordering, but the shape suggests the logic: recorded responses are cheap enough to run on every change yet cannot show how real services respond, which the sandbox queries test, while the deployment gate stays small enough to run every time and the production runs look for problems that appear only after release. The case study also describes failing production traces being handed to an agent that proposes a fix and re-runs the evaluations, with a human reviewing the resulting pull request. It gives no pass rates or counts of problems caught, and it is a vendor's account of a customer, so it shows how Rippling structures its testing, not how well that structure works.