LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA.
LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
Evidence on when LoRAA cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.Full definition → performs like full fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → and when it does not, with recommendations you can apply to your own runs.
