Hong Liu

Liu completed a PhD at Stanford, affiliated with its AI Lab and NLP Group. His research spans synthetic data, multimodal learning and reasoning. At Voyage AI, he led research on representations for finding relevant information: embeddings place content into a space where related items can be compared, while rerankers refine a set of retrieved candidates.[1]

Adapting an optimizer to curvature

Liu co-authored Sophia, an optimizer for language-model pretraining. It periodically estimates the loss function’s diagonal Hessian, measuring local curvature, to scale updates. Clipping bounds each update, while infrequent curvature estimation limits overhead.[2]

Learning multimodal representations

His co-authored MoCa work adapts a vision-language model for embedding tasks through two stages. Continual pretraining jointly reconstructs noisy image and text inputs with bidirectional context. Contrastive fine-tuning then learns from varied multimodal examples, extending beyond simple image-caption pairs. The aim is to represent content for comparison and retrieval rather than only predict the next token.[3]

Training across documents

Synthetic Bootstrapped Pretraining studies relationships between documents instead of treating each as an isolated sequence. A synthesizer learns shared concepts from related material and generates additional training documents, which are used alongside the original corpus. The paper tests this procedure in compute-matched pretraining experiments, separating its effect from simply spending more training compute.[4]