Self-Supervised Hierarchical Alignment for Annotation-Efficient Cross-Modal Retrieval in Medical Imaging

Abstract

Cross-modal retrieval between medical images and radiology reports is critical for clinical decision support but faces significant challenges due to scarce annotated paired data and the need for fine-grained semantic alignment. This research proposes a self-supervised hierarchical alignment framework that reduces annotation dependency by 70% while maintaining competitive retrieval performance. We hypothesize that modality-specific pretext tasks (image inpainting, masked language modeling) combined with hierarchical cross-modal attention will improve fine-grained mAP by 18-22% over global baselines , particularly in zero-shot and few-shot scenarios. Our methodology integrates self-supervised contrastive learning inspired by BYOL, hierarchical region-word attention mechanisms, and knowledge distillation from large teacher transformers to compact student architectures under 50M parameters. Experiments will evaluate MIMIC-CXR and ImageCLEFmed datasets against strong baselines including VSE++ and CLIP, measuring Recall@K, mAP, and inference efficiency (targeting 8-10x speedup) in both standard and zero-shot settings. Expected contributions include: (1) a novel annotation-efficient framework achieving >90% of supervised performance; (2) open-source pre-trained models optimized for medical deployment; and (3) comprehensive empirical guidelines for hierarchical alignment across modalities. This work advances multimodal learning in data-scarce medical domains while enabling efficient clinical deployment.

Keywords