Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
A new preprint investigates how using an external verifier—such as a human or a more capable model—during synthetic data retraining can prevent model collapse. Theoretical analysis in a linear regression setting shows that verifier-guided retraining provides near-term improvements but ultimately converges to the verifier's knowledge center, with gains plateauing unless the verifier is perfectly reliable. Experiments with linear regression, VAEs on MNIST, and SmolLM2-135M on XSUM support these findings.
Why it matters: This work offers a theoretical and empirical framework for understanding and mitigating model collapse in generative AI training, clarifying both the potential and the limitations of synthetic data verification.
Full story at: arXiv Statistical ML ↗