Gisting
- Source
- shopify.engineering
- Author
- Shopify Engineering
- Date
Context
Shopify reports shrinking the repeated instructions for its Sidekick GraphQL from roughly 6,000 tokens to about 1,500 learned replacement tokens, a technique it calls gisting. The replacements are special tokens whose are trained so the model behaves as though it had seen the full prompt, while the model's existing weights stay fixed. Shopify says this was done without losing prediction quality, and that the one-time cost is paid at training.
reuses stored computation for a repeated prompt, but Shopify says it does not remove the cost of generating each answer: every generated token still attends over the whole cached sequence, and reading that cache grows with its length. A shorter sequence reduces that work. In Shopify's load tests at 350 requests per minute, median end-to-end latency fell about 38% and throughput rose 16%, and the company says it could use 14% fewer GPUs for this GraphQL workload. These are results for one Shopify agent, not a general guarantee.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
- system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
Checking sign-in…
Loading comments…