Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
A new preprint systematically compares tokens, bytes, and pixels as language model encodings, controlling for both linguistic content and downstream model capacity. Using parallel sentences in 13 languages, the study traces rate-utility frontiers and finds that no single encoding is optimal across all tasks or capacity regimes. Pixels best preserve surface form, bytes excel at cross-lingual alignment, and tokens are most effective for topic prediction.
Why it matters: This work provides a principled framework for selecting language encodings based on specific tasks, language diversity, and computational constraints, challenging the default reliance on subword tokens.
Full story at: arXiv Computation and Language ↗