MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

Jul 14, 2026

Researchers present a theoretical framework explaining how large language models (LLMs) can store factual knowledge in MLP layers at an information-theoretically optimal rate. They introduce a Transformer-compatible MLP construction that achieves optimal storage capacity scaling and requires 10–104 times fewer parameters than previous methods. The approach supports arbitrary input/output geometries and can be used within Transformer blocks for factual recall tasks. The work also demonstrates modular fact editing by swapping a Transformer's MLP with a new one.

Why it matters: This work advances understanding of how LLMs efficiently store factual knowledge and provides a practical method for more efficient and modular fact storage in Transformer architectures.

Full story at: arXiv Machine Learning ↗