Clinical Trials
Multimodal model predicts 5-year breast cancer recurrence in feasibility study
Chang et alJournal of Imaging Informatics in Medicine

Despite advancements in curative treatments for breast cancer, disease recurrence remains a primary determinant of long-term survival and post-treatment quality of life. While high-dimensional genomic assays offer significant prognostic utility, their implementation is often constrained by high costs and limited accessibility. To address these barriers, researchers developed and evaluated a deep learning-based multimodal integrated learning framework designed to predict 5-year breast cancer recurrence using routinely collected clinical data.
The proposed architecture incorporates three principal stages. First, automated tumor segmentation is executed on ultrasound images using MedSAM to isolate tumor regions and extract detailed radiomics features. Second, unstructured report data are evaluated using a Med-Contrastive Pre-trained Transformers (MedCPT) model guided by predefined, clinically informed queries. Third, image, radiomics, report, and structured clinical features are processed across modality-specific branches—utilizing the Ultrasound Foundation Model (USFM) for imaging data and the FT-Transformer architecture for tabular data. These streams are combined using a mixture-of-experts (MoE)-based classifier trained via a progressive fusion strategy. The model achieved an AUC of 0.7540, accuracy of 74.61%, sensitivity of 70.41%, and specificity of 76.44%, demonstrating the feasibility and additive predictive value of combining multiple routinely collected data types for post-treatment risk stratification.
Key Takeaways
- The final multimodal deep learning model achieved an area under the receiver operating characteristic curve (AUC) of 0.7540, an accuracy of 74.61%, a sensitivity of 70.41%, and a specificity of 76.44% for predicting 5-year breast cancer recurrence.
- Integrating ultrasound imaging, MedSAM-driven tumor radiomics, MedCPT report extraction, and structured tabular clinical data provided additive predictive value over single-modality approaches using routinely available health records.
Read the original sourceJournal of Imaging Informatics in Medicine
