Vibeleaderboard
← All Intel
Intel / article

Shopgym

Source
shopify.engineering
Author
Shopify Engineering
Date
From the Daily Brief

Testing a shopping agent against real stores is hard to repeat. Prices change, inventory moves, and bot detection can block the agent before it does anything useful, so two runs of the same test rarely see the same conditions. Shopify Engineering's ShopGym addresses this by converting live storefronts into self-contained sandbox shops, which Shopify calls ShopArena. It then generates grounded shopping tasks for those sandboxes. Because the sandboxes can be reset, a failure can be replayed exactly, and two versions of an agent can be compared under the same conditions. The same pattern applies beyond commerce: any agent evaluated against a moving live system benefits from snapshotting that system into a resettable environment and generating tasks from the snapshot.

Read the 2026-10-03 Brief →

Context

ShopGym gives shopping agents stores they can revisit under repeatable conditions. Live storefronts change their products, prices, and layouts, making it difficult to tell whether a different result came from the or the website. Shopify's engineering account describes turning storefront observations into anonymized specifications, then using those specifications to generate resettable shops with fictional catalogs.

A second component generates tasks tied to each shop's actual products and controls, and checks that referenced items exist. The reported covered 224 tasks across six shops. Its contribution is a controlled place to compare agents and reproduce failures; using these environments to train more capable shopping agents remains ongoing work.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
  • sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
  • grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Read the source shopify.engineering
More from Shopify Engineering
Recommended reads
Comments

Checking sign-in…

Loading comments…