Attention-Only Transformers Nearly Match Full Transformers When Given Extra Depth
A controlled study pretrains attention-only decoder transformers (Simple Attention Networks, SANs) and compares them to standard transformers, matching for parameters, compute, and depth. While removing feed-forward layers initially reduces performance, reallocating the freed parameter budget to increase attention depth nearly closes the performance gap, with only a 0.006 nat (0.27%) difference in loss at matched parameters. The remaining deficit is mainly in parametric recall, with attention-only models performing better on context-grounded tasks and worse on knowledge-dense queries.
Why it matters: This research demonstrates that feed-forward layers are not strictly necessary for transformer performance if attention depth is increased, potentially simplifying model architectures and informing future design choices.
Full story at: arXiv Machine Learning ↗