CNN ve SVM yöntemleriyle çoklu-odaklı görüntü birleştirmede yeni bir hibrit yaklaşım

Year 2024, Volume: 39 Issue: 2, 1123 - 1136, 30.11.2023

Samet Aymaz

https://doi.org/10.17341/gazimmfd.1208107

Abstract

Çoklu-odaklı görüntü birleştirme, aynı sahnenin farklı odak değerlerine sahip iki veya daha fazla görüntüsünün birleştirilerek tüm-odaklı bir görüntü oluşturulmasıdır. Tüm-odaklı görüntü oluşturulurken temel amaç kaynak görüntülerdeki doğru odak bilgisinin maksimum seviyede birleştirilmiş görüntüye aktarılmasıdır. Önerilen çalışmada, bu amaç doğrultusunda yeni bir hibrit yaklaşım önerilmektedir. Bu yaklaşım, görüntülerden çıkarılan önemli özelliklerin sınıflandırılması ve etkili füzyon kuralları ile birleştirilmesine dayanmaktadır. Özellik çıkarımında, özgün olarak tasarlanan ve basit sistemlerde dahi kolaylıkla çalışabilen bir CNN mimarisi kullanılmaktadır. Çıkarılan özellikler, SVM sınıflandırıcısına verilmekte ve özellik vektörünün odaklı ya da odaksız olarak sınıflandırılması sağlanmaktadır. Sınıflandırma işlemleri sonrasında her bir kaynak görüntü için ikili karar haritaları oluşturulmaktadır. Bu karar haritalarının yanında, önerilen çalışmanın özgün yönlerinden birisi de kararsız bölgelere ait haritaların da çıkarılmasıdır. Bu bölgeler, sınıflandırıcının özellik vektörlerini tam olarak sınıflandıramadığı odaklı bölgelerden odaksız bölgelere geçiş noktalarından oluşmaktadır. Görüntü birleştirmede en önemli konulardan birisi de füzyon kuralının seçimidir. Önerilen çalışmada, sınıflandırıcının kesin olarak karar verebildiği noktalar doğrudan birleştirilmiş görüntüye aktarılırken, kararsız bölgeler için iki alternatif füzyon kuralı kullanılmaktadır. Bunlar gradyan-tabanlı ve laplas-tabanlı füzyon kurallarıdır. Çalışmada her bir füzyon kuralı için, füzyon kurallarının birleştirmeye etkisi gözlemlenmektedir. Sonuç olarak, önerilen çalışmanın performansı objektif performans metrikleriyle değerlendirilmektedir. Sonuçlar incelendiğinde, çalışmanın basit sistemlerde çalışabilen etkili bir füzyon aracı olduğu görülmektedir.

Keywords

CNN, SVM, Özellik çıkarımı, füzyon kuralı, Gradyan.

References

• 1. Akbulut H., Aslantaş V., Multi-exposure image fusion using convolutional neural network, Journal of the Faculty of Engineering and Architecture of Gazi University 38 (3), 1439-1451, 2023.
• 2. Jiang Z., Han D., Chen J., Zhou X., A wavelet based algorithm for multi-focus micro-image fusion, Third International Conference on Image and Graphics (ICIG’04), Hong Kong, China ,176-179, 18-20 Aralık 2004.
• 3. Sujatha K., Punithavathani D. S., Optimized ensemble decision-based multi-focus image fusion using binary genetic Greywolf optimizer in camera sensor networks, Multimedia Tools and Application., 77 (2), 1735–1759, 2018.
• 4. Chen Z., Wang D., Gong S., Zhao F., Application of multi-focus image fusion in visual power patrol inspection, IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC)., Chongqinq, China, 1688–1692, 25-26 Mart 2017.
• 5. Xiao G., Bavirisetti D.P., Liu, G., Zhang, X., Decision-Level Image Fusion. In Image Fusion, Springer, Singapore, 2020.
• 6. Li H., Liu L., Huang W., Yue C., An improved fusion algorithm for infrared and visible images based on multi-scale transform, Infrared Physics and Technology,74, 28–37, 2016.
• 7. Jin X., Hou J., Nie R., Yao S., Zhou D., Jiang Q., He K., A lightweight scheme for multi-focus image fusion, Multimedia Tools and Application, 77 (18), 20286–20302, 2018.
• 8. Petrovic V., Xydeas C., Gradient-based multiresolution image fusion, IEEE Transactions on Image Processing, 13 (2), 228–237, 2004.
• 9. Yang B., Li S., Multi-focus image fusion and restoration with sparse representation, IEEE Transactions on Instrumentation and Measurement, 59 (4) ,884–892, 2010.
• 10. Liu W., Wang Z., A novel multi-focus image fusion method using multi-scale shearing non-local guided averaging filter, Signal Processing, 166, 107252, 2020.
• 11. Li S., Kang X., Hu J., Yang B., Image matting for fusion of multi-focus images in dynamic scenes, Information Fusion, 14 (2), 147–162, 2013.
• 12. Li S., Kwok J., Wang Y., Combination of images with diverse focuses using the spatial frequency, Information Fusion, 2 (3), 169–176, 2001.
• 13. Li M., Cai W., Tan Z., A region-based multi-sensor image fusion scheme using pulse-coupled neural network, Pattern Recognition. Letters, 27 (16), 1948– 1956, 2006.
• 14. Liu Y., Chen X., Peng H., Wang Z., Multi-focus image fusion with a deep convolutional neural network, Information Fusion, 36, 191–207, 2017.
• 15. Tang H., Xiao B., Li W., Wang G., Pixel convolutional neural network for multi-focus image fusion, Information Sciences. 433, 125–141, 2018.
• 16. Amin-Naji M. Aghagolzadeh A. Ezoji, M, Ensemble of CNN for multi-focus image fusion, Information Fusion, 51, 201–214, 2019.
• 17. Ma B., Zhu Y., Yin X., Ban X., Huang H., Mukeshimana, M, Sesf-fuse: An unsupervised deep model for multi-focus image fusion, Neural Computing and Application, 33 (11), 5793–5804, 2020.
• 18. Yan X., Gilani S., Qin H., Mian A., Unsupervised deep multi-focus image fusion, ArXiv, 1806.07272v1, 2018.
• 19. Jung H., Kim Y., Jang H., Unsupervised deep image fusion with structure tensor representations, IEEE Transaction on Image Processing, 29, 3845–3858, 2020.
• 20. Gai D., Shen X., Chen H., Su P., Multi-focus image fusion method based on two-stage of convolutional neural network, Signal Processing, 176, 107681, 2020.
• 21. Zhang Y., Liu Y., Sun P., IFCNN: a general image fusion framework based on convolutional neural network, Information Fusion, 54, 99–118, 2020.
• 22. https://www.cs.toronto.edu/~kriz/cifar.html, Erişim 11 Mayıs 2022.
• 23. https://mansournejati.ece.iut.ac.ir/content/lytro-multi-focus-dataset, Erişim 11 Mayıs 2022.
• 24. https://sites.google.com/view/durgaprasadbavirisetti/datasets?authuser=0, Erişim 11 Mayıs 2022.
• 25. Uğurlu M., Doğru İ.A., Arslan R.S., Detection and classification of darknet traffic using machine learning methods, Journal of the Faculty of Engineering and Architecture of Gazi University, 38 (3), 1439-1451, 2023.
• 26. Sevli O., Diagnosis of diabetes mellitus using various classifiers, Journal of the Faculty of Engineering and Architecture of Gazi University, 38 (2), 989-1001, 2023.
• 27. Qu G., Zhang D., Yan P., Information measure for performance of image fusion, Electronics Letters, 38 (7), 313–315 2002.
• 28. Xydeas C., Petrovic V., Objective image fusion performance measure, Electronics Letters, 36 (4), 308–309, 2000.
• 29. Chen Y., Blum R.S., A new automated quality assessment algorithm for image fusion, Image and Vision Computing, 27 (10), 1421–1432, 2009.
• 30. Eskicioglu A.M., Fisher P.S., Image quality measures and their performance, IEEE Transactions on Communication, 43 (12), 2959–2965 1995.
• 31. Yang C., Zhang J.Q., Wang X. R., Liu, X., A novel similarity based quality metric for image fusion, Information Fusion, 9 (2), 156-160, 2008.
• 32. Liu Y., Liu S., Wang Z., A general framework for image fusion based on multi-scale transform and sparse representation, Information Fusion, 24, 147–164, 2015.
• 33. Ma J., Yu W., Liang P., FusionGAN: a generative adversarial network for infrared and visible image fusion, Information Fusion, 48, 11–26, 2019.
• 34. Bai X., Zhang Y., Zhou F., Quadtree-based multi-focus image fusion using a weighted focusmeasure, Information Fusion 22, 105–118, 2015.
• 35. Jin X., Xi X., Zhou D., Ren X., Yang J., Jiang Q., An unsupervised multi-focus image fusion method based onTransformer and U-Net, IET Image Processing, 17 (3), 733-746, 2022.
• 36. Xu H., Ma J., Jiang J., Guo X., Ling H., U2fusion: A unified unsupervised image fusion network, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (1), 502-518, 2022.
• 37. Bouzos O., Andreadis I., Mitianoudis N., Conditional Random Field-Guided Multi-Focus Image Fusion, Journal of Imaging, 8 (9),240,2022.
• 38. Li B., Peng H.., Wang J., A novel fusion method based on dynamic threshold neural systems and nonsubsampled contourlet transform for multi-modality medical images, Signal Processing, 178, 107793, 2021.
• 39. Tan W., Thitøn W., Xiang P., Zhou H., Multi-modal brain image fusion based on multi-level edge-preserving filtering, Biomed. Signal Process. Control, 64, 102280, 2021.
• 40. Li X., Zhou F., Tan H., Joint image fusion and denoising via three-layer decomposition and sparse representation, Knowl.edge Based Systems, 224, 107087, 2021.
• 41. Zhou D., Jin X., Jiang Q., Cai L., Lee S., Yao S., MCRD-Net: An unsupervised dense network with multi-scale convolutional block attention for multi-focus image fusion, 16 (6), 1558-1574, 2022.
• 42. Wang J., Qu H., Wei Y., Xie M., Xu J., Zhang Z., Multi-focus image fusion based on quad-tree decomposition and edge-weighted focus measure, Signal Processing ,198, 108590, 2022.
• 43. Zhang H., Le Z., Shao Z., Xu H., Ma J., MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion, Information Fusion, 66, 40–53, 2020.
• 44. Liu Y., Liu S.,Wang Z., A general framework for image fusion based on multi-scale transform and sparse representation, Information Fusion, 24 (1), 147–164, 2015.
• 45. Zhang Y., Bai X., Wang T., Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure, Information Fusion, 35, 81–101, 2017.
• 46. Veshki F.G., Vorobyov S.A., Coupled Feature Learning Via Structured Convolutional Sparse Coding for Multimodal Image Fusion. In Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP), Singapore, 2500–2504, 2022.
• 47. Wu K., Mei Y., Multi-focus image fusion based on unsupervised learning, Machine Vision and Applications, 33, 75, 2022.
• 48. Ram Prabhakar K., Sai Srikar V., Venkatesh Babu R., Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs, Proceedings of the IEEE international conference on computer vision, Cambridge, Amerika Birleşik Devletleri, 4714-4722, 6 Ağustos 2002.
• 49. Yin H., Li Y., Chai Y., Liu Z., Zhu Z., A novel sparse-representation-based multi-focus image fusion approach, Neurocomputing, 216, 216–229, 2016.
• 50. Nejati M., Samavi S., Karimi N., Soroushmehr SMR., Shirani S., Roosta I., Najarian K., Surface-are based focus criterion for multi-focus image fusion, Information Fusion, 36, 284–295, 2017.
• 51. Chen C., Gend P., Lu K., Multi-focus image fusion based on multiwavelet and DFB, Chemical Engineering Transactions, 46, 277–283, 2015.
• 52. He K., Zhou D., Zhang X., Nie R., Multi-focus: focused region finding and multi-scale transform for image fusion, Neurocomputing, 320, 157–170, 2018.
• 53. Hua K. L., Wang H. C., Rusdi A. H., Jiang S. Y., A novel multi-focus image fusion based on random walks, Journal of Visual Communication and Image Representation, 25 (5), 951–962, 2014.
• 54. Zhang B., Lu X., Pei H., Liu H., Zhao Y., Zhou W., Multi-focus Image fusion algorithm based on focused region extraction, Neurocomputing, 174, 733–748, 2016.
• 55. Yang Y., Que Y., Huang S., Lin P., Technique for multi-focus image fusion based on fuzzy-adaptiv pulse-coupled neural network, SIVip, 11 (3), 439–446, 2017.
• 56. Liu S., Lu Y., Wang J., Hu S., Zhao J., Zhu Z., A new focus evaluation operator based on max-min filte and its application in high-quality multi-focus image fusion. Multidimensiol System and Signal Processing, 31 (2), 569–590, 2020.
• 57. Jagtab N. S., Thepade S. D., High‑quality image multi‑focus fusion to address ringing and blurring artifacts without loss of information., The Visual Computer, 36, 4353-4371, 2022.
• 58. Li L., Ma H., Jia Z., Si Y., A novel multiscale transform decomposition-based multi-focus image fusion framework. Multimedia Tools and Application, 80 (8), 12389–12409, 2021.
• 59. Wang X., Hua Z., Li J., Multi-focus image fusion framework based on transformer and feedback mechanism, Ain Shams Engineering Journal, 14 (5), 101978, 2022.