Lora

LoRA adapts a model by training a small set of added parameters while leaving its original weights unchanged. Thinking Machines' experiments found that LoRA could match full fine-tuning on small-to-medium instruction and reasoning datasets, provided the adapter had enough capacity for the information in the data. Their supervised comparisons measured prediction loss (log loss) rather than sampling-based evaluations of application performance.

The setup mattered: adapters applied only to the attention layers underperformed adapters that covered the MLP layers, even with a matched number of trainable parameters. Datasets large enough to exceed an adapter's capacity made LoRA underperform, and in some settings large training batches imposed an additional penalty. The findings help explain when LoRA's efficiency can coexist with learning results close to full fine-tuning, and which settings change that comparison. Methods and findings.