To Code Or Not To Code 2024 08 21

Training on code can improve a language model's performance on tasks that do not require writing programs. In a study summarized by Cohere, researchers varied training-data mixtures for models ranging from 470 million to 2.8 billion parameters and compared their performance beyond coding. The publisher abstract reports relative improvements of up to 8.2% on natural-language reasoning and 4.2% on world knowledge when code was added to text-only training.

The authors also report that improving code quality benefited performance across tasks. These findings give model builders a reason to evaluate code as part of general-purpose training data, even when programming is not the intended application. The reported percentages describe the tested settings.