Evaluating the Effectiveness of Panoptic Segmentation Through Comparative Analysis

Abstract

Image segmentation method is extensively used in the fields of computer vision, machine learning, and artificial intelligence. The task of segmentation is to distinguish objects in images either by their boundaries or as entire objects from the entire image. Image segmentation methods are implemented as instance, semantic, and panoptic segmentation. In this article, the panoptic segmentation method, seen as an advanced stage of instance and semantic segmentation, has been applied to three datasets and compared with the instance segmentation method. Experimental results are presented visually. Numerical results have been analyzed with the Panoptic Quality (PQ) and Semantic Quality (SQ) metrics. It has been observed that the segmentation outcome was best for the CityScapes dataset for panoptic segmentation.

Keywords

• Image processing
• image segmentation
• panoptic segmentation
• instance segmentation

References

1. [1] K. Ikeuchi, Computer Vision: A Reference Guide. Cham, Switzerland: Springer International Publishing, 2021.
2. [2] T. Hoeser and C. Kuenzer, “Object detection and image segmentation with deep learning on Earth observation data: A review-part I: Evolution and recent trends,” Remote Sens. (Basel), vol. 12, no. 10, p. 1667, 2020.
3. [3] D. Galea, H.-Y. Ma, W.-Y. Wu, and D. Kobayashi, “Deep learning image segmentation for atmospheric rivers,” Artificial Intelligence for the Earth Systems, 2023.
4. [4] X. Chen et al., “Efficient Decoder and Intermediate Domain for Semantic Segmentation in Adverse Conditions,” Smart Cities, vol. 7, no. 1, pp. 254–276, 2024.
5. [5] J. Yuan, Z. Shi, and S. Chen, “Feature Fusion in Deep-Learning Semantic Image Segmentation: A Survey,” in International Summit Smart City 360°, Cham: Springer International Publishing, 2021, pp. 284–292.
6. [6] P. Garg, A. S. Chakravarthy, M. Mandal, P. Narang, V. Chamola, and M. Guizani, “ISDNet: AI-enabled Instance Segmentation of aerial scenes for smart cities,” ACM Trans. Internet Technol., vol. 21, no. 3, pp. 1–18, 2021.
7. [7] S. A. Güven and M. F. Talu, “Brain MRI high resolution image creation and segmentation with the new GAN method,” Biomedical Signal Processing and Control, vol. 80, 2023
8. [8]Y. Xu, S. Hou, X. Wang, D. Li, ve L. Lu, "A medical image segmentation method based on improved UNet 3+ network," Diagnostics, vol. 13, no. 3, 576, 2023.

9. [9] K. Huang, Y. Zhang, H.-D. Cheng, and P. Xing, “Trustworthy breast ultrasound image semantic segmentation based on fuzzy uncertainty reduction,” Healthcare (Basel), vol. 10, no. 12, p. 2480, 2022.
10. [10] B. Li, Y. Shi, Z. Qi, and Z. Chen, "A survey on semantic segmentation," in 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1233-1240, Nov. 2018.
11. [11] C. Kaymak and A. Ucar, “Semantic image segmentation for autonomous driving using fully convolutional networks,” in 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), 2019.
12. [12] K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biological Cybernetics, vol. 36, no. 4, pp. 193-202, 1980.
13. [13] Y. Lecun, D. Touresky, G. Hinton, and T. Sejnowski, “A theoretical framework for back-propagation,” Proceedings of the 1988 connectionist models summer school, vol. 1, pp. 21–28, 1988.
14. [14] O. Ronneberger, P. Fischer, ve T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III, vol. 18, pp. 234-241, Springer International Publishing, 2015.
15. [15] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, ve A. L. Yuille, "Semantic image segmentation with deep convolutional nets and fully connected crfs," arXiv preprint arXiv:1412.7062, 2014.
16. [16] İ. Kayadibi, U. Köse ve G. E. Güraksın, "Görüntü İşleme Teknikleri ve Evrişimsel Sinir Ağı Kullanılarak Bilgisayar Destekli Diş Segmentasyonu," Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, cilt 1000, no. 1000, ss. 0-0.
17. [17] W. He, X. Wang, L. Wang, Y. Huang, Z. Yang, X. Yao, ... Z. Ge, "Incremental learning for exudate and hemorrhage segmentation on fundus images," in Information Fusion, vol. 73, pp. 157-164, 2021.
18. [18] Y. Zhang, X. Sun, J. Dong, C. Chen, ve Q. Lv, "GPNet: gated pyramid network for semantic segmentation," in Pattern Recognition, vol. 115, 107940, 2021.
19. [19] Q. Sun, Z. Zhang, and P. Li, "Second-order encoding networks for semantic segmentation," Neurocomputing, 2021.
20. [20] W. Mao, J. Zhang, K. Yang, ve R. Stiefelhagen, "Panoptic lintention network: Towards efficient navigational perception for the visually impaired," 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR), s. 857-862, July 2021.
21. [21] Liu, D., Zhang, D., Song, Y., Zhang, F., O’Donnell, L., Huang, H., ... & Cai, W. “Pdam: A panoptic-level feature alignment framework for unsupervised domain adaptive instance segmentation in microscopy images,” IEEE Transactions on Medical Imaging, vol. 40, no. 1, pp.154-165, 2020.
22. [22] S. Albawi, T. A. Mohammed, and S. Al-Zawi, "Understanding of a convolutional neural network," in 2017 International Conference on Engineering and Technology (ICET), pp. 1-6, Aug. 2017.
23. [23] J. Huang, D. Guan, A. Xiao, ve S. Lu, "Cross-view regularization for domain adaptive panoptic segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10133-10144, 2021.
24. [24] Y. Chen, W. Chu, F. Wang, Y. Tai, R. Yi, Z. Gan, ... X. Li, "CFNet: Learning correlation functions for one-stage panoptic segmentation," arXiv preprint arXiv:2201.04796, 2022.
25. [25] G. Narita, T. Seno, T. Ishikawa, ve Y. Kaji, "PanopticFusion: Online volumetric semantic mapping at the level of stuff and things," in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4205-4212, 2019.
26. [26] F. Saeedan and S. Roth, "Boosting monocular depth with panoptic segmentation maps," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3853–3862, 2021.
27. [27] A. Kirillov, K. He, R. Girshick, C. Rother, ve P. Dollár, "Panoptic segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9404-9413, 2019.
28. [28] S. Liu, L. Qi, H. Qin, J. Shi, ve J. Jia, "An End-to-End Network for Panoptic Segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6172-6181, 2019.
29. [29] A. Nivaggioli, J.-F. Hullo, ve G. Thibault, "Using 3D models to generate labels for panoptic segmentation of industrial scenes," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 4, pp. 61-68, 2019.
30. [30] W. Mao, J. Zhang, K. Yang, ve R. Stiefelhagen, "Can we cover navigational perception needs of the visually impaired by panoptic segmentation?," arXiv preprint arXiv:2007.10202, 2020.
31. [31] L. Shao, Y. Tian, ve J. Bohg, "ClusterNet: 3D instance segmentation in RGB-D images," arXiv preprint arXiv:1807.08894, 2018.
32. [32] D. Liu, D. Zhang, Y. Song, H. Huang, ve W. Cai, "Cell R-CNN v3: A novel panoptic paradigm for instance segmentation in biomedical images," arXiv preprint arXiv:2002.06345, 2020.
33. [33] D. Zhang, Y. Song, D. Liu, H. Jia, S. Liu, Y. Xia, ... W. Cai, "Panoptic segmentation with an end-to-end Cell R-CNN for pathology image analysis," in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II, vol. 11, pp. 237-244, Springer International Publishing, 2018.
34. [34] X. Yu, B. Lou, D. Zhang, D. Winkel, N. Arrahmane, M. Diallo, ... A. Kamen, "Deep attentive panoptic model for prostate cancer detection using biparametric MRI scans," in Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part IV, vol. 23, pp. 594-604, Springer International Publishing, 2020.
35. [35] G. Jader, J. Fontineli, M. Ruiz, K. Abdalla, M. Pithon, ve L. Oliveira, "Deep instance segmentation of teeth in panoramic X-ray images," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 400-407, October 2018.
36. [36] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, ve J. Gall, "SemanticKITTI: A dataset for semantic scene understanding of lidar sequences," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9297-9307, 2019.
37. [37] J. Behley, A. Milioto, ve C. Stachniss, "A benchmark for LiDAR-based panoptic segmentation based on KITTI," arXiv preprint arXiv:2003.02371, 2020.
38. [38] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, ve O. Beijbom, "PointPillars: Fast encoders for object detection from point clouds," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, s. 12697-12705, 2019.
39. [39] T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, ... C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, vol. 13, pp. 740-755, Springer International Publishing, 2014.
40. [40] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, ... B. Schiele, "The CityScapes dataset for semantic urban scene understanding," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213-3223, 2016.
41. [41] J. Huang, D. Guan, A. Xiao, ve S. Lu, "Cross-view regularization for domain adaptive panoptic segmentation," arXiv preprint arXiv:2103.02584, 2021.
42. [42] X. Liu, D. Zhao, W. Jia, W. Ji, C. Ruan, Y. Sun, "Cucumber fruits detection in greenhouses based on instance segmentation," IEEE Access, vol. 7, pp. 139635-139642, 2019.
43. [43] M.-C. Roh ve J.-y. Lee, "Refining faster-rcnn for accurate object detection," in 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), IEEE, pp. 514-517, 2017.
44. [44] Y. Ren, C. Zhu, S. Xiao, "Object detection based on fast/faster rcnn employing fully convolutional architectures," Mathematical Problems in Engineering, vol. 2018, 2018.
45. [45] S. Liu, L. Qi, H. Qin, J. Shi, ve J. Jia, "Path aggregation network for instance segmentation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8759-8768, 2018.
46. [46] D. Bolya, C. Zhou, F. Xiao, ve Y. J. Lee, "YOLACT: Real-time instance segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9157-9166, 2019.
47. [47] D. Bolya, C. Zhou, F. Xiao, ve Y. J. Lee, "YOLACT++: Better real-time instance segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.