Faster Retrieval for Better, More Efficient RAG

Apple researchers have shown that the search step behind retrieval-augmented generation (RAG) can be made faster and more accurate when the system knows in advance what kind of questions it will receive. In RAG, a question is turned into a vector, a list of numbers that encodes its meaning, and compared with stored vectors for millions of text passages; the best-matching passage is placed in front of the question before a language model answers. The comparison score is an inner product, so finding the stored vector with the highest score is called maximum inner product search (MIPS). Checking every passage is too slow at that scale, so common systems group the stored vectors into clusters, compare the query only with each cluster's centre point, and then search exhaustively inside the closest clusters. In Apple's research highlight, the researchers point out two weaknesses in that shortcut. It can miss the best passage when that passage sits far from its own cluster's centre but close to the query, and it is designed around the stored passages while ignoring the queries, even though short user questions look quite different from long curated passages and production systems often see a stable, predictable stream of them.

Their fix is to learn the search answer ahead of time. The best score any stored vector can give a query is a known mathematical object, the support function of the stored set, and the gradient of that function at a query (how the best score changes as the query moves) is exactly the best-matching vector. So the team samples queries from the expected distribution, finds each one's true best match once by exhaustive search, and trains small neural networks to reproduce those answers, once and offline, for a fixed collection of passages. One network, SupportNet, estimates for every cluster how good that cluster's best passage is for the query, which replaces the comparison with cluster centres in a standard clustered index (an inverted file index, or IVF). The other, KeyNet, predicts where the best match should lie and hands that predicted vector to an unchanged index in place of the original question, so it works as a query rewriter in front of existing search software such as the FAISS index used in the examples.

The reported results come from Natural Questions, a benchmark that pairs general-knowledge questions with roughly 2.6 million passages, split into 128 clusters, using networks 16 layers deep in four sizes set at 1, 5, 10 and 20 percent of the database's size. The small SupportNet included the cluster holding the true best match among the clusters it searched for 95 percent of queries in 1.74 milliseconds, against 3.36 milliseconds for centre-based routing, and the large one reached 97 percent in the same 1.74 milliseconds; those times include running the network. When each method may scan only 5 percent of the database, the learned routers found the right cluster about 90 to 93 percent of the time, against 87.5 percent for the centre baseline. For KeyNet with the largest model, finding the true best passage for 99 percent of queries took 0.23 milliseconds including the rewrite, against 1.8 milliseconds for the raw question, and scanning 0.001 percent of the database, rewritten queries found the true best passage up to 96.7 percent of the time against 86 percent for raw ones.

These are Apple's own measurements, reported in a summary of the ICML 2026 paper Amortizing Maximum Inner Product Search with Learned Support Functions, which was not reviewed for this explanation. The highlight shows figures for Natural Questions only and says the trend held on the other datasets studied without showing them. It measures whether retrieval finds the exact best match and how quickly, not whether a language model's answers improve, so the "better RAG" in its title is an expected consequence rather than a measured one. The approach also depends on conditions many deployments will not meet. The networks are trained for one fixed set of passages and one expected kind of query, and the highlight claims only that they hold up under mild shifts in what users ask. It does not say how long training takes, what retraining a changing collection would cost, how much memory networks sized at up to a fifth of the database add, or what hardware produced the timings. The result is most relevant to operators of large, stable document collections with predictable questions, and the paper is where those costs would need to be checked.