When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation

Jul 14, 2026

A new preprint reports that, contrary to conventional wisdom, data imbalance can actually promote robust generalization in sufficiently capable models when spurious correlations are present. In a synthetic task, a 2-layer transformer achieved 100% adversarial accuracy in 77% of training runs at a high spurious ratio (0.90), compared to 0% at a balanced ratio (0.50). This effect was not observed in 1-layer models, where data imbalance led models to rely on the shortcut feature instead.

Why it matters: This finding challenges standard assumptions about data balance and suggests new strategies for training models to resist spurious correlations.

Full story at: arXiv Machine Learning ↗