Towards Efficient and Robust Cross-Modal Retrieval: Parameter-Efficient Adaptation of Vision-Language Foundation Models for Scalable Multimedia Search

Abstract

Despite remarkable advances in vision-language foundation models like CLIP, deploying cross-modal retrieval systems at scale remains challenging due to prohibitive computational costs, vulnerability to domain shifts, and strict latency requirements. This proposal introduces a comprehensive framework to simultaneously enhance efficiency, robustness, and scalability through four integrated research thrusts: First, we develop optimized parameter-efficient fine-tuning methods (LoRA, adapters, prompt tuning) that reduce trainable parameters by 98% while preserving >95% of retrieval accuracy. Second, we strengthen model robustness via adversarial training with modality-specific perturbations, targeting <5\% performance degradation on out-of-distribution data compared to >15\% for conventional fine-tuning. Third, we architect a hybrid retrieval pipeline using FAISS approximate search and lightweight neural re-rankers to achieve sub-50ms query latency on 10M-scale datasets with minimal recall loss. Fourth, we systematically characterize efficiency-accuracy-robustness trade-offs across text-to-image, image-to-text, and zero-shot retrieval scenarios. Through extensive experiments on Flickr30K, COCO, and domain-shifted benchmarks, we will evaluate Recall@K, nDCG, latency, and throughput. Our contributions include: (1) Pareto-optimal adaptation recipes with public checkpoints; (2) an open-source robustness evaluation toolkit with 5+ shifted datasets; and (3) actionable guidelines for deploying scalable cross-modal search systems in real-world applications, ultimately bridging the gap between theoretical performance and practical viability.

Keywords