- Category
- AI Tools
- Rank
- No. 1129Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- hkuds
- GitHub
- 572 stars
- Latest release
- 4.38.0.post1+sepllm
- Date
About
ICML 2025 paper accelerating large language models by compressing each segment into a single separator token to speed up inference.
What it does
SepLLM treats punctuation and other separators as memory anchors. It preserves information around those anchors while discarding less useful tokens from the attention cache. The repository supports three modes: adapting existing models without training, training models with sparse attention, and streaming beyond the model’s normal context window.
Why it's ranked here
The core idea has unusually broad implementation coverage. The README reports more than 50 percent KV-cache reduction on GSM8K chain-of-thought with Llama 3 8B and comparable performance, plus streaming beyond four million tokens. Those are compelling results, but the repository demands careful mode selection and version matching.
What's good
Users can test the cache approach without retraining, while researchers can train from scratch for better alignment between training and inference. The training stack supports multiple nodes, accelerated attention kernels, fused operators, evaluation tasks, checkpoint generation, TensorBoard, and Weights & Biases logging. A portable Transformers cache also supports both ordinary and streaming configurations.
Tradeoffs
The three operating modes are not interchangeable. Streaming requires positional encoding shifts and should not be substituted for ordinary evaluation. The bundled Transformers extension targets version 4.38.0 and does not directly support Llama 3.1. The newer portable cache requires Transformers 4.53.x. Optimal downstream performance may require full training, bringing substantial GPU and distributed-system complexity.
How to use it well
SepLLM suits language-model researchers testing cache reduction, sparse attention, long-context streaming, or training and inference consistency. Start with the training-free cache for bounded downstream tasks, then train from scratch when quality warrants the cost. Use the streaming design only beyond pretrained position limits. It does not provide a hosted inference service or a general application framework.
Technical notes+
TrainingFree-SepLLM/demo.py loads SepCache from a modified Transformers package, passes it through past_key_values, and demonstrates Llama 3 8B with FlashAttention 2, bfloat16, and CUDA; it also contains a placeholder Hugging Face login token. Training-SepLLM/train.py enters the Megatron-NeoX pretraining stack, while Training-SepLLM/deepy.py validates SepLLM arguments before delegating launch to DeepSpeed. Training-SepLLM/megatron/training.py constructs SepAttention unless full attention is selected and enforces streaming configuration assertions. Training-SepLLM/eval.py runs an adapted evaluation harness and writes JSON results. Streaming-SepLLM/setup.py exposes a setuptools package, and the README documents the bundled ./package/transformers-4.38.0.post1+sepllm-py3-none-any.whl.
Observed
- Primary language
- Python
- Source licensing
- The supplied training and Megatron-derived source files carry Apache License 2.0 notices.
- Packaging
- Includes a modified Transformers wheel and a setuptools package for the streaming implementation.
- Interfaces
- Python library components plus command-line scripts for training, evaluation, generation, and dataset preparation.
- Compute platform
- The demonstrated inference and training paths use CUDA GPUs; distributed training uses PyTorch and DeepSpeed.
- Model integration
- The bundled Transformers extension supports Meta-Llama-3-8B-Instruct, while the documented portable cache supports the Llama 3.1 series.
Read from README.md, Training-SepLLM/eval.py, Training-SepLLM/deepy.py, Training-SepLLM/train.py, Streaming-SepLLM/setup.py, Training-SepLLM/generate.py, TrainingFree-SepLLM/demo.py, Training-SepLLM/prepare_data.py, Training-SepLLM/megatron/utils.py, Training-SepLLM/megatron/devutil.py, Training-SepLLM/megatron/logging.py, Training-SepLLM/megatron/__init__.py, Training-SepLLM/megatron/training.py, Training-SepLLM/eval_tasks/__init__.py.
What it can do
Compress text segments into separator tokens
Text segments from large language model processing → Compressed separator tokens
Accelerate language model inference speed
Large language model and input text → Faster model predictions with reduced processing time
Process compressed model representations
Separator tokens and model queries → Language model responses with improved efficiency
Optimize memory usage during model inference
Large language model computational requirements → Reduced memory footprint for model operations
Enable faster text generation
Text prompts and generation parameters → Generated text with reduced latency
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.