Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
Gigatoken is an MIT-licensed Rust BPE tokenizer that achieves encoding speeds of 24.53 GB/s on a 144-core AMD EPYC 9565 processor. It outperforms HuggingFace tokenizers by up to 989x and tiktoken by 681x, with speed gains attributed to a hand-written SWAR pretokenizer and pretoken caching rather than a faster BPE merge loop.
Why it matters: This speedup could significantly reduce preprocessing time for large-scale language model training and inference.
Full story at: MarkTechPost / AI ↗