Glossary · inference

prompt caching

Also: context caching

Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.

Agent sessions resend the same large prefix constantly: system prompt, tool definitions, project instructions, conversation so far. Caching stores that processed prefix for a short window, so later calls pay a fraction of the price for the unchanged part.

The consequence for design is that prefix stability is worth money. Putting the volatile parts of a prompt at the end and the stable parts at the front is the difference between hitting the cache on every turn and missing it on every turn, and on a long session that is most of the bill.