Transfuse++: A Hybrid Cnn-Transformer Architecture With Cross-Attention, Temporal Modeling, and Uncertainty Estimation for Medical Image Segmentation Publisher



Noroozi M ; Jamaati S ; Aghapanah H ; Rad AS ; Anar MA ; Darzi A ; Shokouhfar M ; Kazemi HS ; Ghahari M ; Meigoli MSS ; Majidpour J ; Arabi H ; Karimian A
Source: 32nd National and 10th International Iranian Conference on Biomedical Engineering, ICBME 2025 Published:2025

Abstract

Similar to any automated image analysis, precise and credible segmentation is indispensable; however, current convolutional or transformer-based networks alone cannot produce the necessary satisfactory spatial resolution, paired with anatomical global thought and volumetric consistency, in medical practice. We introduce TransFuse++, a promising step that offers incremental improvements to the TransFuse architecture, which closely integrates a CNN branch for local texture representation and a Vision Transformer branch for long-term context encoding. The model is further improved with three new modules: a crossattention mechanism that explicitly aligns the slices, a model with a gated recurrent unit (GRU) that propagates time through stacked sequences, and Monte Carlo dropout that produces pixel-based uncertainty maps. Furthermore, context-dependent cross-skip attention fusion, along with an enhanced BiFusion head, enables more precise boundary delineation without a notable increase in computational cost in the default (no-MC) configuration. When trained in an end-to-end manner on a cardiac X-ray (1,717 frames), a lung CT (6,766 slices) and MR of cirrhotic liver (more than 600 volumes) datasets with an identical 5-fold cross validation protocol, we observe improvements of up to 4.1 percentage points in terms of Dice coefficient and 4.3 percentage points in terms of mIoU, obtaining Dice =0.972 ± 0.002 (X-ray), 0.989 ± 0.001(CT) and 0.923 ± 0.061(MRI). The analysis using Spearman correlation revealed that the per-image Dice coefficients of competing models were either not related to each other (vert rhovert leqslant 0.06, Bonferroni-adjusted p>0.005) or that they introduced complementary specific errors in addition to existing architectures. Ablation experiments validate that cross-attention, temporal modeling, and cross-skipping fusion are key contributors to these incremental improvements. Meanwhile, the uncertainty maps help identify ambiguous areas, making TransFuse++ a promising step toward a confidence-aware segmentation approach for multimodal radiology. © 2025 IEEE.