KANs Enable Auditing in Small Language Models but Show No Consistent Performance Gains Over MLPs
A preprint study investigates Kolmogorov–Arnold Networks (KANs) as replacements for feed-forward networks in small language models. The authors find that small-basis KANs offer a practical and transferable interface for auditing learned scalar transformations, with most edges exhibiting significant nonlinearity and a small fraction inactive. However, across multiple benchmarks and tests, KAN-based architectures do not demonstrate consistent improvements in benchmark accuracy, quality, or latency compared to strong MLP baselines.
Why it matters: This work clarifies the interpretability benefits of KANs while tempering expectations about their performance advantages over established MLP architectures in language modeling.
Full story at: arXiv AI/ML ↗