A batch lane for embedding and reranking, at a third off
- Source
- x.com
- Date
Introducing the Batch API for large-scale async workloads like offline corpus indexing and evals. Key benefits: - Simpler workflows: No queue, retry, and rate limit management with a 12-hour completion window - Higher throughput and limits: Up to 1 GB files, 100K inputs per batch, and 1B tokens per org - Cost savings: 33% discount over the synchronous APIs https://t.co/6evXqDJExY
A simpler and more efficient way to process large embedding and reranking workloads that don’t require real-time responses. Get started with the docs: https://t.co/hV8bqml0cF
Context
Voyage AI's Batch API trades an immediate response for cheaper processing of work that can wait, such as a document collection or running retrieval evaluations. Its documentation describes uploading a file of requests, starting a job and downloading the results later, with a 33% discount relative to standard endpoints.
The service handles request scheduling and retries within a 12-hour completion window. That window is not an unconditional promise that every request succeeds: the documentation says unfinished jobs return their completed results and charge only for the processed requests. The workflow fits offline workloads whose results can be collected asynchronously.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- reranking — A second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…