Shows when to delegate context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →-heavy multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → work from a frontier model to a post-trained open model, using the roughly 40:1 read-to-write token ratio as the cost driver.
Key takeaways · AI-distilled
Because Shopify controls the post-trained weights, it serves these task experts on its own GPUs and decides when they change. Shopify reports that across its post-trained models, self-hosting cuts serving cost by 90-98% compared with frontier model pricing.
Before trying an open model, Shopify tunes the frontier version of the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → with an autoresearch loop that keeps one small prompt, tool-description or control-flow change at a time only if an LLM judge's score improves. That agent becomes the bar to beat.
The judge is calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → to humans: agreement between two product experts on the same conversations sets the target, and Shopify tunes the judge with the GEPA and ACE optimizers until it agrees with held-out human labels as well.
'Self-healing' turns failed production conversations into training data: a frontier critic writes a hint, never the answer, a simulated merchant reruns the request, and passing reruns are anonymized without the hint. It repaired 69% of attempted failures over 86 nights.
Post-trained Nemotron 3.5 Lightning (3B of 30B parameters active) rose from 37.7 to 74.5 on the GraphQL judge after six loops and 61,000 healed examples, versus 73 for the frontier model, with 0.87s median latency against 6.50s and about 90% lower serving cost.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
