Rlhf Can Speak Many Languages Unlocking Multilingual Preference Optimization For Llms 2024 07 05

Preference optimization, a late training stage where a model is pushed toward responses judged better, has mostly been studied in English and Chinese. Cohere's paper on multilingual preference optimization introduces a method for generating multilingual feedback data. According to the full paper, the data mixes completions generated directly in each language with English completions translated into other languages, and preferences between them are labelled by a reward model.

The abstract reports that the preference-trained model achieves a 54.4% win rate against Aya 23 8B, which it calls the state-of-the-art multilingual model in its parameter class, and 69.5% or higher against Gemma-1.1-7B-it, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3, with the work covering 23 languages. A win rate is the share of head-to-head comparisons in which a judge preferred the new model's answer, and the paper's judge was an AI model (GPT-4-Turbo), so this is not a human vote. The abstract also credits cross-lingual transfer and larger datasets, but the paper's results vary by training method, so it should not be read as more languages always helping.