Intel / article

X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding

Source: arxiv.org
Author: Jaeduk Lee, Wan Choi
Date: Published Sep 10, 2026

Context

X-CoSD addresses a communication problem in distributed text generation. A small model on a device drafts candidate text, and a larger server model verifies it. Existing approaches assume matching vocabularies and can require substantial exchanges of probability data between the two.

The paper's enhanced variant lets the server send replacement candidates and their probabilities for checking on the device, reducing the information that must cross the connection. The authors report faster generation and prove that their sampling methods preserve the server model's output distribution. This offers a way to combine differently tokenized models while retaining the larger model's generation behavior. The abstract does not quantify a universal speedup.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.

Key quotes

“we propose cross-vocabulary CoSD (X-CoSD), a lossless and communication-efficient CoSD framework for heterogeneous SLM-LLM vocabularies”

“We prove that both X-CoSD and X-CoSD-E preserve the server LLM distribution, and experiments show that they significantly improve token generation speed while maintaining generation quality comparable to that of the server LLM.”

Read the source arxiv.org