- Category
- AI Tools
- Rank
- No. 102Tools index
- Listed in
- #4 Run models locally
- Type
- APP
- Builder
- lyogavin
- GitHub
- 34.4k stars
- Latest release
- v4.0.0
- Date
About
A library that runs 70B-parameter LLM inference on a single 4GB GPU through aggressive layer-by-layer memory management.
What it does
AirLLM converts model checkpoints into separately stored layers, then loads those pieces only as inference needs them. Sparse mixture-of-experts models can stream individual routed experts instead. The Python interface accepts Hugging Face identifiers or local models and retains a familiar tokenization and generation workflow.
Why it's ranked here
This is a strong specialist tool for fitting unusually large open models onto constrained hardware. Its case rests on a clear mechanism, broad model-family support, optional weight compression, CPU and Apple silicon paths, and concrete memory measurements. Storage demands and disk loading remain important costs.
What's good
The automatic model selection keeps setup compact across Qwen, Llama, DeepSeek, Gemma, Phi, Mistral and other families. Safetensors support permits reading selected tensors instead of entire expert layers. Optional 4-bit and 8-bit weight compression targets the stated disk-loading bottleneck, while prefetching overlaps loading with computation.
Tradeoffs
The first inference decomposes and saves the original checkpoint layer by layer, so substantial disk capacity is still required. The project explicitly identifies disk loading as its main bottleneck. Compression needs an optional dependency, and prefetching is limited to the Llama 2 implementation. Apple support requires Apple silicon, MLX and PyTorch.
How to use it well
Use AirLLM for local experiments where model capacity matters more than loading speed and available GPU memory is the binding constraint. Keep generous checkpoint storage, begin without compression, then test 4-bit or 8-bit storage if loading dominates. It does not remove the need to obtain, store and tokenize the underlying model.
Technical notes+
README.md documents the layer-splitting workflow, Hugging Face and local checkpoint inputs, compression controls, CPU support, and Apple silicon requirements. air_llm/setup.py defines the PyPI package and its core Python dependencies. air_llm/airllm/utils.py implements safetensors-based layer splitting, disk-space checks, optional bitsandbytes compression, hard-link or copy reuse, full-layer loading, and selective tensor loading for MoE experts. air_llm/airllm/__init__.py exposes AutoModel and loads model-specific classes defensively, preserving the generic streaming path when optional family dependencies fail. air_llm/airllm/profiler.py records per-stage timing and minimum free CUDA memory. training/qlora.py and rlhf/qlora_dpo.py provide separate QLoRA and DPO-oriented training scripts rather than the core inference surface.
Observed
- License
- The package metadata classifies AirLLM under the Apache Software License; several bundled training scripts separately identify MIT licensing.
- Primary language
- Python
- Installation
- Published as the airllm Python package and installed with pip.
- Interface
- Python library for loading Hugging Face repository identifiers or local model paths and generating text.
- Core dependencies
- Requires PyTorch, Transformers, Accelerate, Safetensors, Hugging Face Hub, SciPy, SentencePiece and tqdm.
- Optional compression
- Supports 4-bit and 8-bit weight compression through optional bitsandbytes support.
- Platform support
- The repository documents GPU and CPU inference, plus macOS support limited to Apple silicon with MLX and PyTorch.
Read from README.md, requirements.txt, air_llm/setup.py, rlhf/qlora_dpo.py, training/qlora.py, scripts/gen_star_history.py, air_llm/inference_example.py, anima_100k/longer_training.py, anima_100k/modeling_flash_llama.py, scripts/test_cn_dataset_lenghts.py, air_llm/airllm/utils.py, air_llm/airllm/airllm.py, air_llm/airllm/__init__.py, air_llm/airllm/profiler.py.
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.