vramkit

About

A research toolkit for reducing VRAM use in LLM inference. It includes a lossless weight codec with a 2-D context-model range coder, which the README reports gives 33% smaller size than raw 4-bit on Qwen2.5-0.5B weights and beats zstd. It also offers drop-in PyTorch KV-cache quantization that works on ROCm, CUDA, MPS and CPU. Its main finding is that quantizing keys per channel keeps int8 KV near-lossless (+0.1% perplexity at 2x).