ShortOPD: Short-to-Long On-Policy Distillation for Recovering Pruned LLMs

Jul 16, 2026

Structured pruning of large language models (LLMs) often leads to significant degradation in free-form text generation, despite acceptable performance on multiple-choice benchmarks. The ShortOPD method introduces a short-to-long on-policy distillation schedule that detects and truncates repetitive suffixes, focusing training on informative prefixes. This approach achieves up to 9x improvement in generation quality over unrecovered pruned models and matches long-rollout performance using 71% fewer tokens and a quarter of the training time compared to standard recovery methods.

Why it matters: ShortOPD offers a practical solution for efficiently restoring generation quality in compressed LLMs, bridging the gap between pruning research and real-world deployment.

Full story at: arXiv Machine Learning ↗