Powering Billion Scale Vector Search With Opensearch

Searching by meaning becomes an infrastructure problem when the index contains billions of items. In a 2024 prototype, Uber Engineering represented more than 1.5 billion items as numerical vectors and searched them with OpenSearch. A crucial constraint was keeping the search graph, which connects similar vectors, in memory. When the memory allocated was smaller than the graph, queries took tens of seconds because of heavy disk access.

Uber reports reducing 99th-percentile query latency (the response time that 99% of queries beat) from 250 milliseconds to under 120 milliseconds at 2,000 queries per second. Its tuning included setting the shard count equal to the node count and adding replicas, and the account does not say how much each change contributed. It shows how memory capacity and the way work is spread across machines shape search responsiveness in this particular workload.