基于多模态融合的脊柱图像分割方法

doi:10.6040/j.issn.1671-7554.0.2024.0803

摘要/Abstract

摘要： 目的结合脊柱CT和MR多模态医疗图像的互补信息,综合利用骨骼和软组织的详细特征,改善识别的准确性,提高脊柱医疗图像的分割精度,进而提供更全面的脊柱病变评估。方法构建一个多模态医疗图像融合网络模型和一个半监督分割网络模型,分别用于脊柱CT和MR图像的融合以及基于融合图像的分割任务。多模态融合网络通过共享编码器保留不同模态的共同特征,基础编码器提取全局特征,细节编码器专注于局部细节。半监督分割网络模型采用双子网络架构,并引入对比差异评审模块和动态竞争伪标签生成模块来纠正和约束网络训练。结果多模态融合网络在图像信息保留和特征保持方面表现优异,融合图像的高频信息噪声更少。半监督分割网络在Dice系数和Jaccard系数上均表现优异,改善了脊柱软组织与骨组织之间的清晰度。结论多模态医疗图像融合网络和半监督分割网络有效地提升了脊柱图像的融合和分割精度。通过对比差异评审和动态竞争伪标签生成模块的引入,进一步提高分割结果的准确性,为脊柱疾病的评估提供更加清晰和可靠的图像信息。

关键词: 多模态, 图像融合, 图像分割, 半监督, 脊柱

Abstract: Objective By combining the complementary information from spinal CT and MR multimodal medical images, and utilizing detailed features of both bone and soft tissues, to improve the accuracy of identification and enhance the segmentation precision of spinal medical images by soft tissues, thereby providing a more comprehensive assessment of spinal lesions. Methods This paper proposed a multimodal medical image fusion network model for the fusion of spinal CT and MR images and a semi-supervised segmentation network model for the segmentation tasks based on the fused images. The multimodal fusion network retained the shared features of different modalities through a shared encoder, with a basic part extracting global features and a detail part focusing on local details. A dual-network architecture was employed to the segmentation network, which was corrected and constrained by a contrastive difference review module and a dynamic competitive pseudo-label generation module when the network was training. Results The proposed fusion network performed well in preserving image information and features, with less high-frequency noise in the fused images. The semi-supervised segmentation network excelled in both the Dice coefficient and Jaccard index, improving the clarity between spinal soft tissues and bone tissues. Conclusion The proposed multimodal medical image fusion network and semi-supervised segmentation network effectively enhance the fusion and segmentation accuracy of spinal images. The introduction of the contrastive difference review and dynamic competitive pseudo-label generation modules further improved the accuracy of the segmentation results, providing clearer and more reliable image information for the assessment of spinal diseases.

Key words: Multimodal, Image fusion, Image segmentation, Semi-supervised, Spine

中图分类号:

R445.6

代广鑫,王辉,王连雷,刘新宇,张梦华,黄伟杰. 基于多模态融合的脊柱图像分割方法[J]. 山东大学学报 (医学版), 2026, 64(2): 66-77.

DAI Guangxin, WANG Hui, WANG Lianlei, LIU Xinyu, ZHANG Menghua, HUANG Weijie. Spinal images segmentation method based on multimodal fusion[J]. Journal of Shandong University (Health Sciences), 2026, 64(2): 66-77.

参考文献

[1] Li Y, Zheng S, Wu YX, et al. Trends of surgical treatment for spinal degenerative disease in China: a cohort of 37, 897 inpatients from 2003 to 2016[J]. Clin Interv Aging, 2019, 14: 361-366. doi:10.2147/CIA.S191449
[2] Salsali M, Sheikhhoseini R, Sayyadi P, et al. Association between physical activity and body posture: a systematic review and meta-analysis[J]. BMC Public Health, 2023, 23(1): 1670. doi:10.1186/s12889-023-16617-4
[3] Samartzis D, Borthakur A, Belfer I, et al. Novel diagnostic and prognostic methods for disc degeneration and low back pain[J]. Spine J, 2015, 15(9): 1919-1932.
[4] Lenchik L, Heacock L, Weaver AA, et al. Automated segmentation of tissues using CT and MRI: a systematic review[J]. Acad Radiol, 2019, 26(12): 1695-1706.
[5] Martín-Noguerol T, Oñate Miranda M, Amrhein TJ, et al. The role of artificial intelligence in the assessment of the spine and spinal cord[J]. Eur J Radiol, 2023, 161: 110726. doi:10.1016/j.ejrad.2023.110726
[6] Qu B, Cao JP, Qian C, et al. Current development and prospects of deep learning in spine image analysis: a literature review[J]. Quant Imaging Med Surg, 2022, 12(6): 3454-3479.
[7] 马信龙, 马剑雄, 杜育任, 等. 医工结合引领骨科“精” “智” 发展之路[J]. 交通医学, 2019, 33(6): 545-547. MA Xinlong, MA Jianxiong, DU Yuren, et al. The combination of medicine and work leads the development of orthopedics “essence” and “intelligence” [J]. Medical Journal of Communications, 2019, 33(6): 545-547.
[8] Lee S, Jung JY, Mahatthanatrakul A, et al. Artificial intelligence in spinal imaging and patient care: a review of recent advances[J]. Neurospine, 2024, 21(2): 474-486.
[9] Azam MA, Khan KB, Salahuddin S, et al. A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics[J]. Comput Biol Med, 2022, 144: 105253. doi:10.1016/j.compbiomed.2022.105253
[10] 任博文, 韩振川, 吴剑慧, 等. 腰椎椎间融合影像学评价方法的研究进展[J]. 解放军医学院学报, 2022, 43(8): 901-906. REN Bowen, HAN Zhenchuan, WU Jianhui, et al. Research advances in radiographic evaluation methods for lumbar intervertebral fusion[J]. Academic Journal of Chinese PLA Medical School, 2022, 43(8): 901-906.
[11] Zhang YD, Dong ZC, Wang SH, et al. Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation[J]. Inf Fusion, 2020, 64: 149-187. doi:10.1016/j.inffus.2020.07.006
[12] Fedorov A, Beichel R, Kalpathy-Cramer J, et al. 3D slicer as an image computing platform for the Quantitative Imaging Network[J]. Magn Reson Imaging, 2012, 30(9): 1323-1341.
[13] Zamir SW, Arora A, Khan S, et al. Restormer: efficient transformer for high-resolution image restoration[C] //2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). New Orleans, LA, USA: IEEE, 2022: 5718-5729. doi:10.1109/CVPR52688.2022.00564
[14] Fournier Q, Caron GM, Aloise D. A practical survey on faster and lighter transformers[J]. ACM Comput Surv, 2023, 55(14s): 1-40.
[15] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C] //Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015: 234-241.
[16] Abdollahi A, Pradhan B, Alamri A. VNet: an end-to-end fully convolutional neural network for road extraction from high-resolution remote sensing data[J]. IEEE Access, 2020, 8: 179424-179436. doi:10.1109/ACCESS.2020.3026658
[17] Guo P, Xie GQ, Li RF, et al. Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain[J]. Complex Intell Syst, 2023, 9(1): 317-328.
[18] Tang L, Tian CG, Li LD, et al. Perceptual quality assessment for multimodal medical image fusion[J]. Signal Process Image Commun, 2020, 85: 115852. doi:10.1016/j.image.2020.115852
[19] Huang B, Yang F, Yin MX, et al. A review of multimodal medical image fusion techniques[J]. Comput Math Methods Med, 2020, 8279342. doi:10.1155/2020/8279342
[20] Lévêque L, Outtas M, Liu HT, et al. Comparative study of the methodologies used for subjective medical image quality assessment[J]. Phys Med Biol, 2021, 66(15). doi:10.1088/1361-6560/ac1157
[21] Masood RF, Ahmad Taj I, Khan MB, et al. Deep lear-ning based vertebral body segmentation with extraction of spinal measurements and disorder disease classification[J]. Biomed Signal Process Contr, 2022, 71: 103230. doi:10.1016/j.bspc.2021.103230
[22] Liang YW, Fang YT, Lin TC, et al. The quantitative evaluation of automatic segmentation in lumbar magnetic resonance images[J]. Neurospine, 2024, 21(2): 665-675.
[23] Yu LQ, Wang SJ, Li XM, et al. Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[C] //Medical image computing and computer assisted intervention-MICCAI 2019: 22nd international conference, Shenzhen, China, October 13-17, 2019, proceedings, part II 22. Springer International Publishing, 2019: 605-613.
[24] Li SL, Zhang CY, He XM. Shape-aware semi-supervised 3D semantic segmentation for medical images[C] //Medical image computing and computer assisted intervention-MICCAI 2020: 23rd international conference, Lima, Peru, October 4-8, 2020, proceedings, part I 23. Springer International Publishing, 2020: 552-561.
[25] Lou AG, Tawfik K, Yao X, et al. Min-max similarity: a contrastive semi-supervised deep learning network for surgical tools segmentation[J]. IEEE Trans Med Imaging, 2023, 42(10): 2832-2841.
[26] Xu H, Ma JY, Jiang JJ, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Trans Pattern Anal Mach Intell, 2022, 44(1): 502-518.
[27] Tan W, Tiwari P, Pandey HM, et al. Multimodal medical image fusion algorithm in the era of big data[J]. Neural Comput Appl, 2025, 37. doi:10.1007/s00521-020-05173-2
[28] Tran VL, Lin HY, Liu HW. Multitask deep learning for segmentation and lumbosacral spine inspection[J]. IEEE Trans Instrum Meas, 2022, 71: 4005910. doi:10.1109/TIM.2022.3184341
[29] Fu J, Li WS, Du J, et al. Multimodal medical image fusion via Laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy[J]. Comput Biol Med, 2020, 126: 104048. doi:10.1016/j.compbiomed.2020.104048
[30] Muhammad G, Alshehri F, Karray F, et al. A comprehensive survey on multimodal medical signals fusion for smart healthcare systems[J]. Inf Fusion, 2021, 76: 355-375. doi:10.1016/j.inffus.2021.06.007
[31] Dimitri GM, Spasov S, Duggento A, et al. Multimodal and multicontrast image fusion via deep generative models[J]. Inf Fusion, 2022, 88: 146-160. doi:10.1016/j.inffus.2022.07.017
[32] Zhou SK, Greenspan H, Davatzikos C, et al. A review of deep learning in medical imaging: imaging traits, technology trends, case studies with progress highlights, and future promises[J]. Proc IEEE Inst Electr Electron Eng, 2021, 109(5): 820-838.
[33] Das P, Pal C, Acharyya A, et al. Deep neural network for automated simultaneous intervertebral disc(IVDs)identification and segmentation of multi-modal MR images[J]. Comput Methods Programs Biomed, 2021, 205: 106074. doi:10.1016/j.cmpb.2021.106074
[34] Pang SM, Pang CL, Zhao L, et al. SpineParseNet: spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation[J]. IEEE Trans Med Imaging, 2021, 40(1): 262-273.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed