Doc-to-LoRA
https://github.com/sakanaai/doc-to-lora- Category
- AI Tools
- Rank
- No. 784Tools index
Previous survey · No. 779 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- sakanaai
- GitHub
- 809 stars
- Date
About
Hypernetwork from Sakana AI that turns a document into a LoRA, baking factual knowledge directly into model weights.
What it does
Doc-to-LoRA is a research implementation for making a language model answer from supplied context through generated LoRA parameters. Load a pretrained checkpoint, provide a document, then generate responses influenced by that material. Resetting the model removes the internalized information.
Why it's ranked here
The project is compelling because it provides runnable inference, pretrained checkpoints, training scripts, evaluation scripts, and data inspection tools around the paper’s method. It remains a research codebase: setup is dependency-heavy, the documented Python interface handles only one input at a time, and practical use requires downloading model checkpoints.
What's good
The repository covers more than a demo. It includes context-aware training, sequence packing, knowledge-distillation and cross-entropy losses, length-binned evaluation, question-answer scoring, ROUGE, timing instrumentation, and generated-data inspection. The documented reset step also makes comparison against the unmodified model straightforward.
Tradeoffs
The documented Python interface does not support batched inputs. Installation pulls a broad machine-learning stack, including distributed training, quantization, experiment tracking, web interfaces, and evaluation packages. Defaults assume CUDA-oriented execution, bfloat16, flash attention, multiple data workers, and large token limits, so this is not presented as a lightweight local utility.
How to use it well
Use it for reproducing the research, testing whether document-conditioned adapters improve factual answers, or studying context internalization against baseline generation. Start with the pretrained checkpoint and interactive demo before running training experiments. It does not replace document retrieval, indexing, source citation, or a production knowledge-management layer.
Technical notes+
README.md documents a non-batched Python flow that downloads a Hugging Face checkpoint, loads its state dictionary, resets the model, tokenizes a chat prompt, internalizes document text, and generates a response. pyproject.toml defines the ctx-to-lora package for Python 3.10 or newer with setuptools and pins major runtime components including Transformers, DeepSpeed, Accelerate, Datasets, vLLM, and Hugging Face Hub. src/ctx_to_lora/model_loading.py selects causal, bidirectional, or Gemma vision-backed loading paths, supports PEFT wrapping and optional four-bit quantization, and defaults to CUDA with bfloat16. src/ctx_to_lora/configs.py combines YAML and command-line configuration. src/ctx_to_lora/data/packing.py and src/ctx_to_lora/data/collator.py implement packed training inputs plus separate evaluation and generation collation. src/ctx_to_lora/trainer.py provides distillation and cross-entropy trainers with per-context averaging and generated-LoRA regularization. src/ctx_to_lora/metrics.py and src/ctx_to_lora/eval_utils.py cover token accuracy, prefix matching, ROUGE, normalized QA scoring, length groups, and CSV export.
Observed
- Primary language
- Python
- Python requirement
- Python 3.10 or newer
- Packaging
- Setuptools package named ctx-to-lora, configured through pyproject.toml
- Install surface
- uv-based installation through the repository install script
- Interfaces
- Python API, Gradio interactive demo, experiment scripts, and a generated-data viewer
- Model distribution
- Pretrained checkpoints are downloaded from the SakanaAI/doc-to-lora Hugging Face repository
Read from README.md, setup.py, pyproject.toml, src/ctx_to_lora/utils.py, src/ctx_to_lora/configs.py, src/ctx_to_lora/metrics.py, src/ctx_to_lora/pooling.py, src/ctx_to_lora/trainer.py, src/ctx_to_lora/eval_utils.py, src/ctx_to_lora/model_loading.py, src/ctx_to_lora/data/packing.py, src/ctx_to_lora/data/collator.py, src/ctx_to_lora/tracker/timer.py.
What it can do
Convert document to LoRA
Document file → LoRA (Low-Rank Adaptation) file
Extract factual knowledge from document
Document content → Structured knowledge representation
Embed factual information into model weights
Document and base model → Modified model with embedded knowledge
Generate hypernetwork from document content
Document file → Hypernetwork configuration
Create fine-tuned model adapter
Source document and target model → Model adapter with document knowledge
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.