The JEPA Paradox in Language: Why Latent Prediction Fails for Text
A new arXiv preprint argues that Joint-Embedding Predictive Architectures (JEPAs), which have been successful in vision and audio, are fundamentally mismatched to language data due to the lack of conditional concentration in text. The authors formalize this limitation and show, both theoretically and empirically, that applying squared-error latent prediction to text leads to degeneracy and collapse in text encoders, resulting in instability and poor transfer performance. Their experiments with I-JEPA and T-JEPA confirm these predicted failures across multiple runs.
Why it matters: This work highlights a fundamental limitation in applying a popular self-supervised learning approach to language, suggesting that future methods must account for the multiple plausible completions inherent in text.
Full story at: arXiv Computation and Language ↗