ClinCoCoOp: An Interpretable Prompt Learning Framework with Clinical Concept Guidance for Context Optimization

Large Vision-Language Models (VLMs) demonstrate significant potential in representation learning and exhibit strong performance across diverse downstream tasks. Soft prompt learning has emerged as an effective technique for adapting VLMs like CLIP to image classification. However, prevailing prompt learning methods typically generate non-interpretable text tokens, failing to satisfy the stringent interpretability requirements of eXplainable AI (XAI) in high-risk domains such as healthcare. To address this limitation, we introduce a novel interpretable prompt learning framework. Our approach enhances interpretability by incorporating clinical concepts and aligns image semantics with learnable prompts at multiple granularities. Departing from existing methods that apply uniform clinical concept weights across all prompts, we propose two key modules: (1) a Soft-Prompt Clinical Concept Alignment module, which computes image-concept similarity scores to weight clinical concepts before aligning them with the soft prompt (a set of learnable vectors), and (2) a Global-Local Image Soft-Prompt Alignment module, which processes local image regions by incorporating positional encodings and calculating significance weights, complementing the global alignment. Extensive experiments on three medical image datasets (Derm7pt, ISIC2018, Pneumonia) demonstrate the superior classification performance of our method name Clinical Concept CoOp(ClinCoCoOp). Notably, ClinCoCoOp also achieves outstanding zero-shot transfer results on the MED-NODE and ISIC2019 datasets.