KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
KDFlow is a new knowledge distillation framework for large language models that introduces a decoupled architecture, using SGLang for teacher inference and FSDP2 for student training. By transmitting only hidden states via zero-copy data transfer, KDFlow achieves 1.44x to 6.36x speedup over existing frameworks. The system supports both off-policy and on-policy distillation and offers user-friendly APIs for extensibility.
Why it matters: KDFlow enables significantly faster and more efficient distillation of large language models, reducing engineering overhead and accelerating model development.
Full story at: arXiv Computation and Language ↗