FlashTrie: GPU-Accelerated Constrained Beam Search for Generative Retrieval

Jul 14, 2026

FlashTrie is a system that optimizes constrained beam search for generative retrieval tasks on GPUs, featuring a succinct trie layout and a cooperative CUDA kernel to perform decoding entirely on-device. On a library of 800 million keywords with beam widths up to 1000, it reduces trie-search latency to under 3 ms and achieves up to 24x speedup over a highly optimized multi-threaded CPU baseline. In a large-scale online A/B experiment on a commercial search engine, FlashTrie delivered a statistically significant +0.71% revenue lift.

Why it matters: FlashTrie enables real-time, large-scale constrained decoding for generative retrieval, directly improving commercial search engine performance and revenue.

Full story at: arXiv Machine Learning ↗