Compound Sparsity Framework Outperforms Single-Dimensional Compression for LLMs
A new preprint introduces a compound sparsity framework for large language models (LLMs) that combines static parameter pruning (via low-rank approximation and channel pruning) with dynamic token-level layer skipping. Experimental results show that this approach consistently outperforms single-mechanism compression methods at equivalent total sparsity, delaying performance degradation on language understanding tasks and maintaining stronger modeling performance. The study also finds that a near-balanced allocation between parameter and token-level sparsity is most effective under a fixed sparsity budget.
Why it matters: This work demonstrates that integrating multiple compression strategies can significantly improve LLM efficiency, offering a practical route to more compact models with less performance loss.
Full story at: arXiv Machine Learning ↗