Multimodal Deep Learning Frameworks for Breast Cancer Detection Using Ultrasound, Mammography, and Clinical Data Publisher



Rashedi S ; Rashedi A ; Shahreza BO ; Tabatabaeian M ; Abedi I
Source: Informatics in Medicine Unlocked Published:2026

Abstract

Background Accurate differentiation between benign and malignant breast lesions remains challenging when relying on a single imaging modality. Multimodal deep learning offers the potential to integrate complementary diagnostic information from mammography, ultrasound, and clinical data to improve classification performance . Methods A prospective dataset of 92 biopsy-confirmed patients with paired mammography and ultrasound images was analyzed. Four multimodal deep learning models—CUF-MT, AMW-CNN, MM-ABMIL, and CNN-LSTM—were developed using five-fold cross-validation and evaluated on an internal test set. Performance was assessed using AUC, accuracy, sensitivity, specificity, precision, and MCC. Additional analyses included ablation studies, missing-modality experiments, and statistical comparison using McNemar's test. Results Multimodal models consistently outperformed unimodal approaches. AMW-CNN achieved the highest cross-validation performance (AUC = 0.992), while CUF-MT demonstrated the best generalization on the internal test set (AUC = 0.91). McNemar's test revealed statistically significant differences between models, with AMW-CNN outperforming others in most pairwise comparisons. Missing-modality analysis showed that ultrasound was the dominant contributor to classification performance, whereas mammography primarily improved prediction calibration. Age provided complementary information, supported by a large effect size (Cohen's d = 2.94). Ablation studies confirmed the importance of adaptive fusion and cross-modal interactions. Conclusion Multimodal deep learning enhances breast lesion classification, with different architectures offering complementary strengths in performance and generalization. The proposed framework highlights the importance of modality-aware evaluation and robustness analysis, supporting the development of clinically reliable decision-support systems. © 2026 The Authors.