LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
Establishes that LoRAA cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.Full definition → matches full fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → far more broadly than the literature suggested, provided it is applied to all layers and the learning rate is set relative to FullFT, with capacity rather than rank alone as the limit.
