How closely LoRA tracks full fine-tuning, measured
- Source
- x.com
- Date
LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA.
Context
Customizing a model does not always require changing all its weights. keeps the original weights fixed and trains compact additions, reducing training memory requirements. In Thinking Machines' experiments, this often matched full on small-to-medium instruction and reasoning datasets when the adapters had sufficient capacity.
The conditions matter. Applying adapters across the model worked better than restricting them to layers, and larger datasets could exceed the adapters' capacity. The supervised comparisons primarily measured prediction loss, a measure of how well the model predicts the training text. The findings help developers choose an adaptation method and configuration without assuming that a smaller update must sacrifice accuracy. Methods and findings.
- LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Checking sign-in…
Loading comments…