ReportMedSAM: Guiding Medical Image Segmentation Through Radiology Reports

Jul 17, 2026

ReportMedSAM introduces a framework for medical image segmentation that leverages free-form radiology reports by replacing discrete extraction with a learnable concept bank. The system aligns organ-level embeddings with clinical corpora using contrastive learning and employs a frozen medical vision-language encoder. It dynamically activates task-specific Mixture-of-Experts modules based on report content, enabling robust segmentation and easy extension to new tasks without retraining existing components. Evaluation on the AbdomenAtlas 3.0 dataset shows competitive segmentation accuracy and effective handling of linguistic variability in clinical reports.

Why it matters: This approach advances scalable and robust medical image segmentation directly from natural language reports, addressing limitations of prior rule-based or phrase-matching methods.

Full story at: arXiv Computation and Language ↗