Shitao Xiao researches text embeddings and retrieval models. He is first author of RetroMAE and C-Pack and a coauthor of M3-Embedding, work that connects model pretraining with finding relevant passages across languages and document lengths.[1][2][3][4]
Start with RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder →
Links
GitHub Contributions
Train a representation for retrieval
RetroMAE pretrains a sentence encoder by giving the encoder and decoder differently masked versions of a sentence. The decoder reconstructs the original using the encoded representation and its own masked input. A full encoder and a shallow decoder create an asymmetric task that encourages the representation to carry useful information. The paper evaluates the resulting models on dense-retrieval benchmarks.[2]
Several retrieval methods in one model
M3-Embedding supports dense retrieval, sparse retrieval and multi-vector retrieval, with inputs ranging from short sentences to long documents. Its self-knowledge distillation combines relevance scores from these retrieval functions into a training signal. That lets the model learn from different ways of matching a query and document, rather than optimizing each function in isolation.[3]
Package the resources needed for evaluation
C-Pack brings together resources for general Chinese embeddings, including training and evaluation material. Xiao’s public portfolio connects these retrieval projects with the FlagEmbedding ecosystem. The contribution includes data, models and testing infrastructure, giving researchers a way to inspect the pipeline around an embedding model.[1][4]
Tools, writing & talks
Writing, research & talks
Start here
An asymmetric masking and reconstruction method for retrieval-oriented representations.
Read the encoder-and-decoder diagram to see what the masking task teaches. Basic understanding of embeddings and masked-language modeling helps with the training details.
First author[2]- M3-Embeddingarxiv.org
A multilingual model combining dense, sparse and multi-vector retrieval training.
Coauthor[3] Data, models and benchmarks for general Chinese text embeddings.
First author[4]