- Category
- AI Tools
- Rank
- No. 622Tools index
Previous survey · No. 615 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 1.2k stars
- Latest release
- v0.5.4
- Date
About
KV-cache compression toolkit for LLMs — drop-in techniques to cut memory and extend context length.
What it does
KVPress lets researchers compare ways to discard, reshape, or preserve selected transformer cache entries. It plugs into Hugging Face Transformers, compresses a context during prefill, then reuses that compressed context when answering one or more questions. Experimental support also compresses periodically during generation.
Why it's ranked here
This is a strong research toolkit because it combines many published approaches behind one abstraction, includes repeatable evaluation machinery, and offers a familiar Transformers pipeline. Its value lies in comparison and experimentation, not in hiding every operational detail or providing a complete serving system.
What's good
The method selection is unusually broad, spanning importance scoring, chunk selection, per-layer budgets, head-wise allocation, cache merging, and composable wrappers. The supplied pipeline handles tokenization and chat templates, while the evaluation framework registers multiple long-context and reasoning datasets. Optional evaluation and FlashAttention dependencies keep those heavier requirements separate from the base installation.
Tradeoffs
The custom pipeline uses greedy decoding, so sampling workflows need direct model generation. Decoding compression remains experimental and accepts only scorer-based methods, with no support for multiple questions in that mode. Some approaches require multiple forward passes, while one temporarily creates roughly twice the context-length cache during prefill. Importing the package also modifies Transformers attention handlers globally.
How to use it well
Use KVPress when researching cache policies, benchmarking long-context models, or testing several questions against one compressed context. Start with the registered Transformers pipeline, compare methods through the evaluation harness, then move to direct model generation when sampling controls matter. It does not provide a hosted API, deployment platform, or general model-serving layer.
Technical notes+
pyproject.toml defines a Python 3.10+ Hatchling package with Torch, Transformers, Datasets, Accelerate, PEFT, Pandas, and NumPy dependencies, plus eval and flash-attn extras. kvpress/__init__.py exports the press classes and calls patch_attention_functions() during import. kvpress/attention_patch.py wraps every entry in Transformers’ ALL_ATTENTION_FUNCTIONS to support head-specific masking. kvpress/pipeline.py implements KVPressTextGenerationPipeline, using a DynamicCache, context-only prefill compression, cache reuse across questions, and greedy token decoding. evaluation/evaluate_registry.py centralizes dataset, scorer, and press registrations. kvzap/data.py and kvzap/train.py provide data collection and surrogate-model training for KVzap.
Observed
- License
- Apache-2.0
- Primary language
- Python
- Install surface
- Published package installable with pip; local environments use uv
- Build system
- Hatchling
- Interface
- Python library with a registered Hugging Face Transformers pipeline
- Python support
- Python 3.10 or newer
- Optional dependencies
- Separate extras for evaluation tooling and FlashAttention
- Test automation
- Makefile runs pytest with coverage and JUnit reports against a tests directory
Read from README.md, Makefile, pyproject.toml, kvzap/data.py, kvzap/train.py, kvpress/utils.py, kvzap/__init__.py, kvpress/__init__.py, kvpress/pipeline.py, evaluation/__init__.py, evaluation/evaluate.py, kvzap/evaluate_aime.py, kvpress/attention_patch.py, evaluation/evaluate_registry.py, kvpress/presses/__init__.py.
What it can do
Compress KV-cache memory usage
LLM model with KV-cache → Compressed KV-cache with reduced memory footprint
Extend context length capacity
LLM model with limited context window → LLM model with extended context length capabilities
Apply drop-in compression techniques
Existing LLM implementation → Optimized LLM with integrated compression
Reduce memory requirements for inference
Memory-intensive LLM operations → Memory-optimized inference pipeline
Optimize KV-cache storage efficiency
Uncompressed key-value cache data → Compressed and optimized cache storage
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.