Vibeleaderboard
← All Intel
Intel / article

Perplexity Open-Sources Multimodal Late-Interaction Embedding Models

Source
perplexity.ai
Author
Perplexity
Date

Context

Most search squeezes each document into one vector, so a long or visually dense page has to fit everything a future query might need into a single point. Perplexity's pplx-embed-v2-late models keep a 128-dimensional vector for every token instead and score with MaxSim: each query token is matched to its most similar document token, and those best matches are summed. Different parts of a question can then match different parts of a page. Like ordinary dense embeddings, documents are still encoded once, ahead of time, without seeing the query, which is what separates this late-interaction approach from slower cross-encoders that read query and document together. The price is storage and scoring cost that grows with document length, which Perplexity says calls for a different indexing and serving setup than one-.

The same models also embed images and rendered pages directly, so PDFs, slides and scans can be searched without first running OCR and discarding tables, figures and layout. The main test for that is ViDoRe V3, a benchmark of about 26,000 pages from 10 professional document collections with 3,099 human-verified queries in six languages; its authors found that visual retrievers beat text-based ones and that late-interaction models substantially improve results. On the benchmark's public subset (two of the ten datasets are held back as a private test set), Perplexity reports 65.2% for the 9B model, behind only Tencent's EVIE among the baselines it compared, and 62.3% for the 0.6B model. Perplexity stresses storage here: its models emit 128 numbers per token, against 2,048 or 4,096 for the larger competing multi-vector models.

The less familiar part is the shared embedding space. Perplexity trained an 18B teacher and it into the 9B and 0.6B students by aligning each student's per-token output with the teacher's, so vectors from either model can be compared. A collection indexed once with the 9B model can then be searched with queries encoded by the 0.6B model, which uses roughly 240M active parameters for text. Against using the 0.6B model on both sides, that raised ViDoRe V3 from 62.3% to 63.5% and the 72 domain-specific text tasks by 1.6 points on average, with no added query-time cost. Perplexity says this recovers about half the text-quality gap to running 9B on both sides.

The other headline numbers are Perplexity's own evaluations, with a full technical report promised later this year. Q2D-Web, where the 9B and 0.6B models reach 74.8% and 73.6% Recall@1000 against 69.3% for the previous best, is Perplexity's own web-retrieval benchmark built from production search queries. The company reports the judgment set that adds labels because its other two sets come from what its own systems already surfaced and would favour models that rank the same way. It also says it left benchmark-related datasets out of training to avoid inflating scores.

Terms in this piece · Glossary
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • Vector search — Finding content by similarity between embeddings.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • Large language model — A model trained on large amounts of data to process and generate language.
Read the source www.perplexity.ai
More from Perplexity
Recommended reads
Comments

Checking sign-in…

Loading comments…