When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
A new preprint demonstrates that log-likelihood-based (perplexity) evaluation can significantly underestimate the performance gap between teacher and distilled student models in autoregressive generation tasks. The authors introduce GenDistill, a multi-stage pipeline for distilling Transformers into efficient Hybrid Kimi Delta Attention models, and systematically analyze the impact of various design choices. They find that dataset selection, completion-only masking, and freezing attention layers are particularly important for maintaining generation quality in distilled models.
Why it matters: This work exposes limitations of perplexity-based evaluation in model distillation and offers practical insights for building efficient, high-quality generative models.
Full story at: arXiv Computation and Language ↗