SpecLA: Efficient Speculative Decoding for Linear-Attention Models
SpecLA is a speculative decoding runtime designed for stateful linear-attention models. It introduces topology-aware kernels, compact factor recovery, and confidence pruning to efficiently verify draft tokens, addressing the unique challenges of recurrent dependencies in these models. On an NVIDIA H100 with a GDN-1.3B target, SpecLA achieves up to 1.70x end-to-end speedup over standard autoregressive decoding.
Why it matters: This work enables faster inference for linear-attention models, which are important for long-context tasks, by overcoming inefficiencies in traditional autoregressive decoding.
Full story at: arXiv Computation and Language ↗