Text-to-LoRA
https://github.com/sakanaai/text-to-lora- Category
- AI Tools
- Rank
- No. 1115Tools index
Previous survey · No. 1102 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- sakanaai
- GitHub
- 1.3k stars
- Date
About
Hypernetworks that generate task-specific LoRA adapters for LLMs from a textual task description.
What it does
Text-to-LoRA converts a written task brief into adapter weights for a local language model. It embeds the description, encodes that task signal, produces low-rank weight updates across model layers, and applies or saves the resulting adapter for inference and evaluation.
Why it's ranked here
This is a serious research implementation with trained checkpoints, generation and evaluation flows, and two documented training methods. Its reported benchmark tables show consistent gains over base models and competitive multi-task adapter baselines across three model families. The substantial hardware and setup demands keep it from being a casual utility.
What's good
The project covers the full experimental loop: checkpoint download, local generation, adapter export, evaluation, supervised training, reconstruction training, and asynchronous checkpoint selection. It supports Mistral, Llama, and Gemma examples. Aligned descriptions perform noticeably better overall, showing that the text input carries useful task information despite reasonable results from random descriptions.
Tradeoffs
The demos require downloaded checkpoints and more than 16GB of GPU memory when both models run together. Training is expensive, taking about five days per model on one H100, while reconstruction first trains hundreds of oracle adapters. Installation includes a hardware-specific FlashAttention wheel. Evaluation can vary between runs because the documented vLLM setup is nondeterministic with LoRA.
How to use it well
Use it for research on rapid task adaptation when you have suitable GPU hardware and want adapters that can be evaluated separately from generation. Start with the supplied checkpoints and clear, aligned task descriptions, then compare against the base model and multi-task adapters. It does not replace hosted inference, low-resource fine-tuning, or a lightweight production API.
Technical notes+
pyproject.toml defines a Python package built with setuptools and dependencies including Transformers, PEFT, Accelerate, Gradio, vLLM, and Weights & Biases. src/hyper_llm_modulator/hyper_modulator.py contains task encoders and the HyperModulator that generates LoRA weights. src/hyper_llm_modulator/hooks.py injects generated low-rank updates through PyTorch forward hooks. src/hyper_llm_modulator/sft_trainer.py trains from task-conditioned examples, while src/hyper_llm_modulator/recon_trainer.py learns against oracle adapter weights. src/hyper_llm_modulator/vllm_eval.py evaluates adapters through vLLM and FishFarm tasks. README.md documents the CLI, local Gradio UI, training scripts, checkpoint watcher, and reproducibility caveats.
Observed
- Primary language
- Python
- Packaging
- Setuptools project declared in pyproject.toml, with uv used for environment and dependency installation
- Interfaces
- Command-line generation and evaluation scripts, plus a local Gradio web UI
- Runtime requirement
- Python 3.10 or newer
- Model examples
- Documented flows cover Mistral-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-2-2b-it
- Hardware surface
- The documented demos require more than 16GB of GPU memory when loading both models
- Training modes
- Supervised fine-tuning and reconstruction from oracle LoRA adapters
Read from README.md, setup.py, pyproject.toml, src/hyper_llm_modulator/data.py, src/hyper_llm_modulator/hooks.py, src/hyper_llm_modulator/configs.py, src/hyper_llm_modulator/vllm_eval.py, src/hyper_llm_modulator/lora_mixing.py, src/hyper_llm_modulator/sft_trainer.py, src/hyper_llm_modulator/recon_trainer.py, src/hyper_llm_modulator/res_aggregator.py, src/hyper_llm_modulator/hyper_modulator.py.
What it can do
Generate LoRA adapter from text description
Textual task description → Task-specific LoRA adapter
Create specialized model weights for specific tasks
Natural language task specification → Custom LoRA weights
Adapt LLM behavior for custom use cases
Task description and target LLM → Fine-tuned model adapter
Automatically configure hypernetwork parameters
Task requirements in text format → Optimized hypernetwork configuration
Generate domain-specific model adaptations
Domain description and task objectives → Domain-adapted LoRA module
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.