- Category
- AI Tools
- Rank
- No. 640Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- microsoft
- GitHub
- 13.8k stars
- Latest release
- RoBERTa-large
- Date
About
Microsoft's reference implementation of Low-Rank Adaptation — efficient fine-tuning that trains a tiny number of parameters per task.
What it does
LoRA swaps selected PyTorch layers for compatible variants that add two low-rank matrices beside the existing weights. Training freezes the original model and updates those added matrices. For deployment, the learned update can merge into the base weight, avoiding an extra inference path.
Why it's ranked here
The case is strong because the repository pairs a compact library with reproducible language understanding and generation examples. Its reported experiments train 0.8 million parameters instead of 125 million for RoBERTa base, while producing comparable aggregate GLUE results. GPT-2 experiments also compare favorably with full tuning, adapters, and prefix tuning.
What's good
The integration model stays close to ordinary PyTorch training. Developers replace chosen layers, freeze everything except adaptation parameters, and save adaptation-only checkpoints. It handles embeddings, convolutions, ordinary linear projections, and combined projections where only selected slices should adapt. Evaluation mode can merge updates into existing weights, so inference adds no latency.
Tradeoffs
Support is limited to PyTorch and a small set of layer families. Choosing which projections, embeddings, or multilayer perceptron layers to adapt remains task-specific experimentation. Combined query, key, and value projections need special handling. Adaptation checkpoints cannot run alone because they still require the original pretrained checkpoint, and loading requires relaxed key matching.
How to use it well
Use it when adapting large PyTorch models for several tasks without storing a complete model copy per task. Start with query and value projections, measure task quality, then test embeddings or multilayer perceptron layers when useful. Save only adaptation weights and load them after the base checkpoint. It does not supply pretrained models or replace model evaluation and configuration search.
Technical notes+
loralib/layers.py defines Embedding, Linear, MergedLinear, and ConvLoRA, with low-rank A and B parameters, alpha-over-rank scaling, optional dropout, frozen base weights, and train/eval merge transitions. loralib/utils.py provides mark_only_lora_as_trainable and lora_state_dict, including optional handling for all biases or only biases attached to adapted layers. setup.py packages loralib for Python 3.6 or newer. README.md documents loading the pretrained state first, loading LoRA state with strict=False, and reproducing results through examples/NLG/ and examples/NLU/.
Observed
- License
- MIT License
- Primary language
- Python
- Install surface
- Python package installable as loralib through pip or directly from the GitHub repository
- Interface
- Importable PyTorch library
- Framework support
- PyTorch only
- Supported layer families
- Linear, embedding, two-dimensional convolution, and merged linear projections
- Example coverage
- Includes language generation examples for GPT-2 and language understanding examples for RoBERTa and DeBERTa
Read from README.md, setup.py, loralib/utils.py, loralib/layers.py, loralib/__init__.py, examples/NLU/setup.py, examples/NLU/hubconf.py, examples/NLU/src/transformers/__init__.py, examples/NLU/src/transformers/data/__init__.py, examples/NLU/src/transformers/utils/__init__.py, examples/NLU/examples/legacy/seq2seq/__init__.py, examples/NLU/src/transformers/models/__init__.py, examples/NLU/src/transformers/commands/__init__.py.
What it can do
Fine-tune pre-trained models with low-rank adaptation
Pre-trained model and training dataset → Fine-tuned model with task-specific adaptations
Train model adaptations with reduced parameter count
Base model and task-specific data → Low-rank adaptation weights
Apply LoRA adaptations to existing models
Base model and LoRA weight files → Adapted model for specific tasks
Create task-specific model variants
Foundation model and task training data → Specialized model for target task
Merge multiple LoRA adaptations
Base model and multiple LoRA weight sets → Combined model with multiple task capabilities
Export LoRA weights separately from base model
Fine-tuned LoRA model → Standalone LoRA weight files
Intel on LoRA
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.