The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
Source
arxiv.org
Author
Connor Makowski, Willem Guter
Date
Context
The Functionalizer represents spelling variations through a shared base form plus reversible instructions for features such as capitalization, accents and repeated characters. This lets a tokenizer preserve the original text while using fewer distinct vocabulary entries. Across six text and code collections, the authors report vocabulary reductions of up to 16%. The tradeoff is that a smaller vocabulary does not always mean shorter input sequences: indentation-heavy code became more compact, while natural-language prose became longer. The method therefore offers a domain-dependent representation choice, with downstream model tests still limited to a small scale.
Key takeaways · AI-distilled
Standard subword tokenizers treat capitalization and accent variants like hello, Hello, HELLO, and Héllo as separate vocabulary entries, fragmenting the embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → space; the Functionalizer instead factors these into a base tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → plus reversible transform codes.
The Functionalizer encodes casing, 13 diacritic types, and character repetition as parametric opcodes in the Unicode Private Use Area, fully reversible back to the original text, avoiding lossy normalization.
Across six corpora, the method cut actual vocabulary slot requirements by up to 16% while still achieving complete coverage of the text.
The effect on sequence length is domain-dependent: it shortens indentation-heavy code sequences but lengthens natural-language prose sequences.
On preliminary tests with 25M-parameter GPT-2 scale models, the method markedly improved code syntax validity and code-character perplexity while keeping prose coherence similar to standard tokenization; the authors call for production-scale validation.
Terms in this piece · Glossary
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleBeyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI SystemsIvan Dobrovolskyi
articletokenizers v1: encode, decode and scaling, measuredhuggingface.co
articleTEFM: Token-Efficient Faithful Modeling for Structured DataZhichao Hou, Lingdao Sha, Xueyu Mao, Yang Liu, Peijie Qiu, Rui Song