Multi-level Context Modeling Improves Expert Selection in Mixture-of-Experts Models
A new framework, MCF-MOE, is introduced to enhance expert selection in Mixture-of-Experts (MoE) models by fusing cross-layer semantic information and local token-level interactions. The approach addresses context incompleteness in existing MoE routers, leading to more consistent and context-aware expert routing. Experimental results on language modeling and understanding tasks show that MCF-MOE achieves improved routing consistency and better downstream performance compared to strong MoE baselines.
Why it matters: This work offers a practical advance in MoE model design, potentially enabling more efficient and stable large-scale Transformer systems by improving expert routing consistency.
Full story at: arXiv Computation and Language ↗