LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
Researchers introduce LookME, a framework that enables lookup-based enhancement for multimodal embeddings in vision-language models (VLMs). LookME employs a hierarchical two-level lookup method and a sparse injection strategy to efficiently retrieve and inject relevant multimodal embeddings, supporting partitioned storage and on-demand loading. Experimental results indicate that LookME outperforms text-only PLE-style methods on multiple visual benchmarks, demonstrating improved efficiency and performance.
Why it matters: LookME provides a memory-efficient approach to scaling vision-language models, potentially enabling their deployment in resource-constrained environments.
Full story at: arXiv Computer Vision ↗