Introducing the Batch API for large-scale async workloads like offline corpus indexing and evals. Key benefits:
- Simpler workflows: No queue, retry, and rate limit management with a 12-hour completion window
- Higher throughput and limits: Up to 1 GB files, 100K inputs per batch, and 1B tokens per org
- Cost savings: 33% discount over the synchronous APIs
https://t.co/6evXqDJExY
A simpler and more efficient way to process large embedding and reranking workloads that don’t require real-time responses.
Get started with the docs: https://t.co/hV8bqml0cF
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
reranking — A second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Bulk corpus indexing and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → runs no longer need your own queue, retry, and rate-limit handling, and cost 33% less than the synchronous endpoints.
