Cyber Chaos Monkey

Shopify's security team now runs AI agents that attack its own live production systems, and it keeps them from doing damage in two ways. Each attacking agent is boxed in by network and credential limits, and a second agent running on a different model watches everything it does and can shut it down. Shopify calls the system Cyber Chaos Monkey, after the Netflix tool that randomly switched off production servers to prove its services could survive failures. The reasoning is economic. Shopify says frontier models are making vulnerability discovery cheap for everyone, and its bug bounty program has received almost 300% more reports since 2024. Its bet is that the same models, combined with knowledge no outsider has (source code, past bug reports, threat intelligence and deployment records), let it find the most damaging bugs first. The system extends an earlier Shopify harness that reviewed code and proved findings with tests; the new step is attacking the running site itself, using test shops and organizations.

Each attack, which Shopify calls an engagement, starts from an idea supplied by a human security engineer, another security agent or an earlier run. It can be narrow, such as looking for flaws similar to one reported through the bug bounty, or broad, such as gaining the ability to run code on the Shopify admin. One component, Discovery, uses models to find places in the code or configuration that might be exploitable. Promising leads go to a sandbox where agents try to exploit them. The technique Shopify singles out as most effective is sending a live request to Shopify, reading the production logs to see how it was handled, checking that against the source code and trying again. Reports flow back to Discovery, which can launch further engagements.

The containment is conventional. Every engagement runs as its own Kubernetes job, the attacking agent has no elevated permissions, and it receives only short-lived Shopify credentials, such as a session cookie and CSRF token, that it can read but not change. Keys for internal services and model providers are inserted by a proxy, so the agent never holds them, and outbound traffic can reach only Shopify production domains, those services, model providers and a few package sources. The less familiar layer governs behavior. Written rules of engagement let the agent try any exploit against its target but forbid things like moving into other systems or keeping access once an exploit works. Shopify does not trust the attacking agent to obey them, so a separately prompted judge agent on a different model reads its whole session in real time, including its reasoning and tool calls, and can kill it. The design fails closed: if the judge times out, crashes or stops ruling, the run ends, and because both agents share one Kubernetes pod, an eviction stops them together.

The evidence for how well this works is Shopify's own summary: tens of thousands of engagements and dozens of impactful findings that it acted on, without an outage or incident. The post does not say what those findings were, how severe they were, how often the judge has stopped a run or whether anyone has tested if an attacking agent can hide a rule violation from it. The fail-closed design covers a judge that stops working, not one that keeps running and misses something. Outside red-teaming of Anthropic's internal agent monitoring, published by METR, found new weaknesses in that kind of oversight, which is the sort of test this post does not report for its judge. Shopify has described a careful design and reports a clean record so far; the post reports no outside check of either.