Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs

A recommendation model can learn to speak in both product identifiers and ordinary language. In Eugene Yan's prototype, product descriptions are converted into short learned codes called semantic IDs. Similar products can share parts of a code, giving the identifiers information that arbitrary catalog numbers lack.

Yan adds these codes to a language model's vocabulary and trains it on catalog information and user interaction sequences. The written examples show conversations that revise recommendations by adding a preference or constraint. This demonstrates a possible conversational interface to a learned catalog; the prototype's limited training and examples leave broad reliability and recommendation accuracy unresolved.

“The idea is simple: Instead of using random hash IDs for videos or songs or products, we can use semantically meaningful tokens that an LLM can natively understand.”

Eugene Yan

“The result is a language model that can converse in both English and item IDs, not with retrieval or other tools, but as a single, “bilingual” model where items (i.e., semantic IDs) are part of its vocabulary.”

Eugene Yan

“While this LLM-recommender hybrid may not match the raw precision of a specialized multi-stage recsys, it offers a new capability: steerability and reasoning on recommendations”

Eugene Yan

“In my experiments with a three-level codebook, with each level having 256 codes, we saw collisions on ~10% of the 66k products.”

Eugene Yan

“The trained RQ-VAE achieved 89% unique semantic IDs across 66k products on the three quantization levels.”

Eugene Yan