Lightweight Depthwise Convolution Improves LLMs with Minimal Parameter Increase
A new preprint demonstrates that adding lightweight depthwise convolution to Qwen3 Transformer blocks can improve average accuracy across seven benchmarks, while increasing model parameters by less than 0.01%. The most effective configuration applies convolution to the projected queries, keys, and values before attention, using a kernel size of 3 and no extra normalization. The method enhances the model's sensitivity to local context, particularly for repeated token IDs.
Why it matters: This approach provides a simple and highly parameter-efficient way to introduce local inductive bias into large language models, potentially improving their performance with negligible computational overhead.
Full story at: arXiv Computation and Language ↗