Brain-Encoding Model's Predicted Responses Outperform Visual Backbone for Video Memorability on One Dataset, Not Another
A preprint study evaluates the TRIBE v2 brain-encoding model's predicted fMRI responses as features for forecasting video memorability. On the VideoMem dataset, these brain-encoded features outperform the model's own visual backbone, while on Memento10k, the backbone performs better. The dataset-specific advantage is robust to controls for sample size and feature compression, and a vision-orthogonal memorability signal is localized to the ventral occipito-temporal cortex.
Why it matters: The findings suggest that brain-encoding models can capture human-relevant behavioral signals missed by standard vision models, but their utility depends on the dataset, underscoring the importance of diverse benchmark evaluation.
Full story at: arXiv Computer Vision ↗