Vibeleaderboard
← All Intel
Intel / video

Build Hour: Prompt Caching

Source
youtube.com
Author
OpenAI
Date

Context

A solutions engineer at OpenAI explains in a Build Hour on prompt caching that caching reuses work already done on the start of a prompt. When requests share an identical beginning, the system skips reprocessing it. It applies automatically from 1,024 , in 128-token steps, and a hit needs the same content in the same order, so stable material such as system instructions should come first.

The presenter says discounts on cached input grew from 50% to 75% to 90% across successive model families. In a test of 2,300 prompts between 1,024 and 200,000 tokens, caching made time to first output about 67% faster for the longest prompts. The cache holds a large set of floating-point numbers from the model's computation, the presenter says, not the text itself. Default retention is five to ten minutes, with an option up to 24 hours. These are OpenAI's figures as stated in the session.

Terms in this piece · Glossary
  • prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Read the source www.youtube.com
More from OpenAI
Recommended reads
Comments

Checking sign-in…

Loading comments…