- Category
- AI Tools
- Rank
- No. 352Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- microsoft
- GitHub
- 6.6k stars
- Latest release
- v0.2.2
- Date
About
Microsoft's prompt and KV-cache compression for LLMs — up to 20x compression with minimal accuracy loss for cheaper, faster inference.
What it does
LLMLingua is a Python library that uses a smaller language model to decide which prompt tokens carry essential information, then removes the rest before a larger model sees them. Its methods cover long-context prompts, task-agnostic token classification, structured JSON, and safety-focused compression research.
Why it's ranked here
The case is unusually concrete: three documented compression methods, published research references, runnable notebooks, and integrations with Prompt flow, LangChain, and LlamaIndex. The repository reports better long-context retrieval and faster task-agnostic compression, while exposing enough controls to test those claims against a real workload.
What's good
The library supports target token budgets, compression rates, multiple filtering levels, protected tokens, and structured JSON fields. It installs through pip and includes examples for RAG, meetings, reasoning, code, and LlamaIndex. CPU, CUDA, and Apple silicon device selections appear in the model-loading logic.
Tradeoffs
The default configuration targets CUDA and a 7B model, so lightweight deployment is not the starting point. Compression can produce visibly fragmented language, as the quick-start output demonstrates. Reported quality gains are workload-specific research results, so teams still need task-level evaluation before trusting compressed prompts in production.
How to use it well
Use it when long prompts, retrieval context, or repeated model calls make token budgets consequential. Start with the task-agnostic compressor, preserve digits and critical terms, then compare downstream answers against uncompressed baselines. It compresses input context; it does not provide retrieval, model serving, or a complete application framework.
Technical notes+
setup.py packages llmlingua for Python 3.8 or newer with Transformers, Accelerate, PyTorch, tiktoken, NLTK, and NumPy. llmlingua/__init__.py exports PromptCompressor, while llmlingua/prompt_compressor.py loads Hugging Face causal or token-classification models and implements prompt, structured JSON, context, sentence, and token filtering paths. llmlingua/utils.py handles token boundaries, deterministic seeding, and JSON markup. experiments/llmlingua2/evaluation/compress.py supplies a dataset-oriented CLI. The Makefile runs pytest against ./tests/, but its install target uses dist/sdtools*, which does not match the package name declared in setup.py.
Observed
- License
- MIT License
- Primary language
- Python
- Installation
- Published package surface documented as pip install llmlingua
- Interface
- Python library, plus an argparse-based experimental compression CLI
- Python support
- Python 3.8 or newer
- Model runtime
- Hugging Face Transformers with PyTorch and Accelerate
- Device handling
- Model loader recognizes CUDA, CPU, and MPS device selections
- Testing surface
- Makefile defines pytest execution against a tests directory
Read from README.md, Makefile, setup.py, setup.cfg, pyproject.toml, llmlingua/utils.py, llmlingua/version.py, llmlingua/__init__.py, llmlingua/prompt_compressor.py, experiments/securitylingua/filter.py, experiments/securitylingua/label_word.py, experiments/securitylingua/train_roberta.py, experiments/llmlingua2/evaluation/utils.py, experiments/llmlingua2/evaluation/metrics.py, experiments/llmlingua2/evaluation/compress.py.
What it can do
Compress LLM prompts
Original text prompts → Compressed prompts with up to 20x size reduction
Compress KV-cache data
Key-value cache from LLM inference → Compressed KV-cache with reduced memory footprint
Reduce inference costs
Standard LLM queries and prompts → Cost-optimized inference with maintained accuracy
Accelerate LLM response times
LLM inference requests → Faster inference results through compression
Optimize prompt efficiency
Verbose or lengthy prompts → Streamlined prompts with preserved semantic meaning
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.