DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search
Source
Rapha\"el Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Am\'elie Chatelain
Author
Rapha\"el Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Am\'elie Chatelain
Date
Key takeaways · AI-distilled
Two 149M-parameter retrieval models trained on the same open data behave differently: the dense single-vector DenseOn scores 56.20 average nDCG@10 on the BEIR benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → suite, the ColBERT-style late-interaction LateOn 57.22 — a new best for that size class.
The multilingual models share a backbone, data, and objective, yet split on generalization: the dense model is strong on English and the eight translated languages but falls off outside them, while the late-interaction model holds up on languages and scripts it never saw.
That split points at a mechanism: matching query and document token-by-token, instead of squeezing each into one vector, is what lets translate-train stretch past the languages you actually translated into.
The recipe is fully reconstructible: 665M English contrastive pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → pairs curated down from 1.4B across 34 public sources, then 1.88M supervised pairs with mined hard negatives, with models, data, and training code released.
Terms in this piece · Glossary
determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
An open training recipe for retrieval models lets teams build code and long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → search without depending on closed data.
Key quotes
“State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap.”
“They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class.”
“Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts.”
“This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe.”
