The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
Source
Connor Makowski, Willem Guter
Author
Connor Makowski, Willem Guter
Date
Why it matters
A reversible pre-tokenizer encodes casing, diacritics and repetition as prefix opcodes instead of separate vocabulary entries, shrinking required vocabulary slots by up to 16% across code and natural-language corpora without losing information.
articleBeyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI SystemsIvan Dobrovolskyi
articleTEFM: Token-Efficient Faithful Modeling for Structured DataZhichao Hou, Lingdao Sha, Xueyu Mao, Yang Liu, Peijie Qiu, Rui Song
articleFunction-Level Execution Feedback for Code Preference OptimizationIdris Nechnech, Sehwan Kim, Jimin Seo, Yeongoon Kim, Minhae Oh, Sangwoo Hong, Jungwoo Lee