Sparse Inter-Layer Dependencies of Transformer FFN Neurons

Jul 15, 2026

A new training-free attribution method is introduced to analyze the dependencies of feedforward network (FFN) neurons in Transformer models. The study finds that small, sparse subsets of upstream activations and attention outputs can preserve neuron activations with high fidelity, even when other inputs are masked. This reveals that FFNs, despite their dense parameterization, have sparse and structured inter-layer dependencies at the neuron level. The method is scalable and can be used for circuit-level interpretability and identifying sparse pathways for potential efficiency gains.

Why it matters: This work offers a practical and scalable approach to understanding and potentially optimizing the internal structure of Transformer models, which could lead to more efficient inference and improved interpretability.

Full story at: arXiv Machine Learning ↗