A 3D U-NET BASED ON EARLY FUSION MODEL: IMPROVEMENT, COMPARATIVE ANALYSIS WITH STATE-OF-THE-ART MODELS AND FINE-TUNING

Abstract

Multi-organ segmentation is the process of identifying and separating multiple organs in medical images. This segmentation allows for the detection of structural abnormalities by examining the morphological structure of organs. Carrying out the process quickly and precisely has become an important issue in today's conditions. In recent years, researchers have used various technologies for the automatic segmentation of multiple organs. In this study, improvements were made to increase the multi-organ segmentation performance of the 3D U-Net based fusion model combining HSV and grayscale color spaces and compared with state-of-the-art models. Training and testing were performed on the MICCAI 2015 dataset published at Vanderbilt University, which contains 3D abdominal CT images in NIfTI format. The model's performance was evaluated using the Dice similarity coefficient. In the tests, the liver organ showed the highest Dice score. Considering the average Dice score of all organs, and comparing it with other models, it has been observed that the fusion approach model yields promising results.

Keywords

• Computed Tomograph
• Multi Organ Segmentation
• Deep Learning
• Fusion Model
• U-Net

References

1. N. Shen et al., "Multi-organ segmentation network for abdominal CT images based on spatial attention and deformable convolution," Expert Systems with Applications, vol. 211, p. 118625, 2023.
2. Y. Wang, Y. Zhou, W. Shen, S. Park, E. K. Fishman, and A. L. Yuille, "Abdominal multi-organ segmentation with organ-attention networks and statistical fusion," Medical image analysis, vol. 55, pp. 88-102, 2019.
3. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
4. A. Şeker, B. Diri, and H. H. Balık, "A review about deep learning methods and applications," Gazi J Eng Sci, vol. 3, no. 3, pp. 47-64, 2017.
5. G. Guo and N. Zhang, "A survey on deep learning based face recognition," Computer vision and image understanding, vol. 189, p. 102805, 2019.
6. H.-S. Bae, H.-J. Lee, and S.-G. Lee, "Voice recognition based on adaptive MFCC and deep learning," in 2016 IEEE 11th Conference on Industrial Electronics and Applications (ICIEA), 2016: IEEE, pp. 1542-1546.
7. S. Caldera, A. Rassau, and D. Chai, "Review of deep learning methods in robotic grasp detection," Multimodal Technologies and Interaction, vol. 2, no. 3, p. 57, 2018.
8. G. Litjens et al., "A survey on deep learning in medical image analysis," Medical image analysis, vol. 42, pp. 60-88, 2017.

9. M. Toğaçar and B. Ergen, "Biyomedikal Görüntülerde Derin Öğrenme ile Mevcut Yöntemlerin Kıyaslanması," Fırat Üniversitesi Mühendislik Bilimleri Dergisi, vol. 31, no. 1, pp. 109-121, 2019.
10. H. R. Roth et al., "A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation," in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11, 2018: Springer, pp. 417-425.
11. T. Ghosh, L. Li, and J. Chakareski, "Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images," in 2018 25th IEEE International Conference on Image Processing (ICIP), 2018: IEEE, pp. 3034-3038.
12. B. Kayhan, "Deep learning based multiple organ segmentation in computed tomography images," Master's thesis, Konya Technical University, 2022.
13. B. Kayhan and S. A. Uymaz, "Multi Organ Segmentation in Medical Image.," Current Studies in Healthcare and Technology pp. 59-72, 2023.
14. R. Trullo, C. Petitjean, D. Nie, D. Shen, and S. Ruan, "Joint segmentation of multiple thoracic organs in CT images with two collaborative deep architectures," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3, 2017: Springer, pp. 21-29.
15. M. Larsson, Y. Zhang, and F. Kahl, "Robust abdominal organ segmentation using regional convolutional neural networks," Applied Soft Computing, vol. 70, pp. 465-471, 2018.
16. H. R. Roth et al., "Deep learning and its application to medical image segmentation," Medical Imaging Technology, vol. 36, no. 2, pp. 63-71, 2018.
17. C. Shen et al., "On the influence of Dice loss function in multi-class organ segmentation of abdominal CT using 3D fully convolutional networks," arXiv preprint arXiv:1801.05912, 2018.
18. H. Kakeya, T. Okada, and Y. Oshiro, "3D U-JAPA-Net: mixture of convolutional networks for abdominal multi-organ CT segmentation," in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11, 2018: Springer, pp. 426-433.
19. S. Vesal, N. Ravikumar, and A. Maier, "A 2D dilated residual U-Net for multi-organ segmentation in thoracic CT," arXiv preprint arXiv:1905.07710, 2019.
20. O. Mietzner and A. Mastmeyer, "Automatic multi-object organ detection and segmentation in abdominal CT-data," medRxiv, p. 2020.03. 17.20036053, 2020.
21. B. Rister, D. Yi, K. Shivakumar, T. Nobashi, and D. L. Rubin, "CT-ORG, a new dataset for multiple organ segmentation in computed tomography," Scientific Data, vol. 7, no. 1, p. 381, 2020.
22. Y. Liu et al., "CT‐based multi‐organ segmentation using a 3D self‐attention U‐net network for pancreatic radiotherapy," Medical physics, vol. 47, no. 9, pp. 4316-4324, 2020.
23. X. Fang and P. Yan, "Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction," IEEE Transactions on Medical Imaging, vol. 39, no. 11, pp. 3619-3629, 2020.
24. F. Zhang, Y. Wang, and H. Yang, "Efficient context-aware network for abdominal multi-organ segmentation," arXiv preprint arXiv:2109.10601, 2021.
25. H. Kaur, N. Kaur, and N. Neeru, "Evolution of multiorgan segmentation techniques from traditional to deep learning in abdominal CT images–A systematic review," Displays, vol. 73, p. 102223, 2022.
26. Z. Xu. "Multi-atlas labeling beyond the cranial vault - workshop and challenge." https://www.synapse.org/#!Synapse:syn3193805/wiki/217760. (accessed 6.3.2021).
27. X. Gao, Y. Qian, and A. Gao, "COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer models," arXiv preprint arXiv:2107.01682, 2021.
28. V. Chernov, J. Alander, and V. Bochko, "Integer-based accurate conversion between RGB and HSV color spaces," Computers & Electrical Engineering, vol. 46, pp. 328-337, 2015.
29. L. Wang, C. Wang, Z. Sun, and S. Chen, "An improved dice loss for pneumothorax segmentation by mining the information of negative areas," IEEE Access, vol. 8, pp. 167939-167949, 2020.
30. H. Cao et al., "Swin-unet: Unet-like pure transformer for medical image segmentation," in European conference on computer vision, 2022: Springer, pp. 205-218.
31. J. Chen et al., "Transunet: Transformers make strong encoders for medical image segmentation," arXiv preprint arXiv:2102.04306, 2021.
32. G. Xu, X. Wu, X. Zhang, and X. He, "Levit-unet: Make faster encoders with transformer for medical image segmentation," arXiv preprint arXiv:2107.08623, 2021.
33. X. Huang, Z. Deng, D. Li, and X. Yuan, "Missformer: An effective medical image segmentation transformer," arXiv preprint arXiv:2109.07162, 2021.
34. Y. Xie, J. Zhang, C. Shen, and Y. Xia, "Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation," in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, 2021: Springer, pp. 171-180.
35. H.-Y. Zhou, J. Guo, Y. Zhang, L. Yu, L. Wang, and Y. Yu, "nnformer: Interleaved transformer for volumetric segmentation," arXiv preprint arXiv:2109.03201, 2021.
36. F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation," Nature methods, vol. 18, no. 2, pp. 203-211, 2021.
37. A. Hatamizadeh et al., "Unetr: Transformers for 3d medical image segmentation," in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2022, pp. 574-584.
38. A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. R. Roth, and D. Xu, "Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images," in International MICCAI Brainlesion Workshop, 2021: Springer, pp. 272-284.