Breaking the 1.58-bit Barrier for Ternary LLMs

Intel researchers studied how ternary large language models, which restrict weights to negative one, zero, or positive one, actually use those three values in practice, and found that zero accounts for up to 51.5% of weights across the models they examined. Standard ternary packing schemes such as five trit encoding treat all three values as equally likely, which wastes storage and bandwidth on a value that dominates the distribution. The team built BITCOS, a layout that separates a bitmap marking which weights are zero from a sign vector for the nonzero ones, exploiting that skew directly. Tested across 29 models, BITCOS beat standard five trit packing on 26 of them, delivering up to 1.27 times faster GPU decode throughput along with gains on CPU inference. Ternary models are already attractive for their small memory footprint, and this result shows that the storage format itself, not just the bit width, has meaningful room for improvement. For teams deploying quantized models at the low end of precision, the technique is a concrete, hardware validated way to extract more speed from ternary weights without retraining or changing the model's architecture.