Damaged Mosaic Image Inpainting By Using Generative Adversarial Network

Abstract

Mosaics, one of the oldest known works of art, have been developed and used by many different civilizations throughout history. Destruction is frequently encountered in mosaic works that have survived from the past to the present. Artifacts can be damaged due to the natural conditions, the negative effects of people or the nature of the objects. The necessity of repairing the damage in these artifacts and reaching their original appearance is the basic need of mosaic artifacts as in all historical artifacts. Image inpainting problem is a current problem that is tried to be solved with different techniques in the literature. In this study, the results of the image inpainting problem on the mosaic data set with deep learning-based methods were examined. Image inpainting architecture is used with contextual attention to correct missing regions in the mosaic image. Comparative results of this architecture with different adversarial generator network architectures were examined using the same data set. Model was retrained with the mosaic dataset using learning transfer. It was observed that the index of structural similarity between the original image and the repaired image in the tested mosaic samples ranged from 0.92 - 0.95 in lightly damaged images and between 0.72 - 0.89 in heavily damaged images, according to the damage ratio. With the implemented image inpainting model, high success was achieved in image inpainting in mosaic paintings with little damage.

Keywords

• Mosaic
• Image inpainting
• deep learning
• generative adversarial networks

Çekişmeli Üretici Ağlar Kullanılarak Hasarlı Mozaik Görüntülerinin Tamamlanması

Abstract

Bilinen en eski sanat eserlerinden olan mozaikler tarih boyunca çok farklı uygarlıklar tarafından geliştirilmiş ve kullanılmışlardır. Geçmişten günümüze ulaşan mozaik eserlerinde tahribat sıklıkla rastlanmaktadır. Gerçekleşen doğa koşulları, insanların olumsuz etkileri veya nesnelerin doğası gereği yıpranmasından dolayı tahribata uğrayan eserler olabilmektedir. Bu eserlerdeki tahribatın onarılması ve orijinal görüntüsüne ulaşılması gerekliliği tüm tarih eserlerinde olduğu gibi mozaik eserlerinin de temel ihtiyacıdır. Görüntü tamamlama problemi literatürde farklı teknikler ile çözülmeye çalışılan güncel bir problemdir. Bu çalışmada görüntü tamamlama problemini derin öğrenme tabanlı yöntemlerle mozaik veri seti üzerindeki sonuçları incelenmiştir. Mozaik görüntüsündeki eksik bölgelerin düzeltilmesi bağlamsal dikkat ile görüntü tamamlama mimarisi kullanılmıştır. Bu mimari aynı veri seti kullanılarak farklı çekişmeli üretici ağ mimariler ile karşılaştırılmalı sonuçları incelenmiştir. Öğrenme aktarımı kullanılarak mozaik veri seti ile yeniden model eğitilmiştir. Test edilen mozaik örneklerdeki orijinal görüntü ile hasarı giderilmiş görüntü arasındaki yapısal benzerlik indisinin yapılan hasar oranına göre az hasarlı görüntülerde 0.92 - 0.95 çok hasarlı görüntülerde ise 0.72 - 0.89 arasında olduğu gözlemlenmiştir. Gerçekleştirilen görüntü tamamlama modeli ile az hasarlı mozaik resimlerinde görüntü tamamlamada yüksek başarı elde edilmiştir.

Keywords

• Mozaik
• görüntü tamamlama
• derin öğrenme
• çekişmeli üretici ağlar

References

1. Arjovsky M, Chintala S, Bottou L, 2017. Wasserstein Gan. arXiv 2017. arXiv preprint arXiv:.07875, 30(4). Barnes C, Shechtman E, Finkelstein A, Goldman D. B, 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Transactions on Graphics, 28(3), 24.
2. Bassier C, 1974. Weiterentwicklung der Konservierungsmethoden für Mosaiken. Arbeitsblätter für Restauratoren(7 1), 43-52.
3. Ballester C., Bertalmio M., Caselles V., Sapiro G., Verdera J., 2001, Filling-in by joint interpolation of vector fields and gray levels, 10(8):1200–1211.
4. Bertalmio M, Vese L, Sapiro G, Osher S, 2003. Simultaneous Structure and Texture Image Inpainting. IEEE Transactions on Image Processing, 12(8), 882-889.
5. Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A., 2020. Albumentations: Fast and Flexible Image Augmentations. Information, 11(2), 125.
6. Criminisi A., Perez P., Toyama K., 2003. Object removal by exemplar-based inpainting. In Computer Vision and Pattern Recognition, 2003 IEEE Computer Society Conference on, volume 2, pages II– II. IEEE.
7. Efros A. A, Freeman W. T, 2001. Image Quilting for Texture Synthesis and Transfer. 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, August 12-17, 2001.
8. Efros A. A., Leung T. K, 1999. Texture Synthesis by Non-Parametric Sampling. Seventh IEEE International Conference on Computer Vision, Kerkyra, September 20-27, 1999.

9. Eskici B, 1997. Taş Eserlerin Korunması Üzerine Notlar. Türk Arkeoloji Dergisi, 31, 338-392.
10. Goodfellow I, Bengio Y, Courville A, 2016. Deep Learning: MIT Press, United States.
11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S and Bengio Y, 2014. Generative adversarial nets. In Advances in Neural 114 Information Processing Systems (NIPS), 8-13 December 2014; Montréal/Canada
12. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A. C, 2017. Improved Training of Wasserstein Gans. Advances in Neural Information Processing Systems, 30.
13. Higgins, I., Matthey, L., Glorot, X., Pal, A., Uria, B., Blundell, C., Lerchner, A., 2016. Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv:1606.05579.
14. Hore A, Ziou D, 2010. Image Quality Metrics: PSNR vs. SSIM. 20th International Conference on Pattern Recognition, İstanbul, August 23-26, 2010.
15. Iizuka S, Simo-Serra E, Ishikawa H, 2017. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics, 36(4), 1-14.
16. Kaya U., Yılmaz A., 2019. Derin Öğrenme, 1-2, ISBN:978-605-2118-399.
17. Langr J, Bok V, 2019. GANs in Action (MEAP Edition Ed.). Manning Publications, United States.
18. Levin A., Zomet A., Weiss Y,2003. Learning how to inpaint from global image statistics. In null, page 305. IEEE.
19. Li Y, Liu S, Yang J, Yang M.H, 2017. Generative Face Completion. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, July 21-26, 2017.
20. Ma Y., Liu X., Bai S., Wang L., He D., Liu A., 2019. Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, August 2019.
21. Makantasis K., Karantzalos K., Doulamis A.,Doulamis N., 2015. Deep supervised learning for hyperspectral data classification through convolutional neural networks. In Geoscience and Remote Sensing Symposium , 2015 IEEE International, pp. 4959- 4962.
22. Ogan A, Mirmiroğlu V, 1955. Kaariye Camii Eski Hora Manastiri. Türk Tarih Kurumu Yayınları,6.
23. Pacal I., 2022, Deep Learning Approaches for Classification of Breast Cancer in Ultrasound (US) Images, Journal of the Institute of Science and Technology, 12(4): 1917 – 1927.
24. Pacal I., Karaman A., Karaboga D., Akay B., Basturk A., Nalbantoglu U., Coskun S., 2022. An efficient real-time colonic polyp detection with YOLO algorithms trained by using negative samples and large datasets, Computers in Biology and Medicine, 141(September 2021):105031.
25. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros A. A, 2016. Context Encoders: Feature Learning by Inpainting. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, June 27-30, 2016.
26. Papernot N., Abadi M., Erlingsson U., Goodfellow I., Talwar K., 2016. Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755.
27. Radford A, Metz L, Chintala S, 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv Preprint arXiv:.06434.
28. Silva T. A, 2022. Beginner’s Guide to Generative Adversarial Networks (GANs), https://wiki.pathmind.com/generative-adversarial-network-gan (Erişim Tarihi: 16.09.2022).
29. Toğaçar M., Ergen B.,2019. Biyomedikal Görüntülerde Derin Öğrenme ile Mevcut Yöntemlerin Kıyaslanması, Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 31(1), 109-121.
30. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang T S, 2018. Generative Image Inpainting with Contextual Attention. IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, June 18-23, 2018.
31. Weiss, K., Khoshgoftaar, T. M., & Wang, D., 2016. A survey of transfer learning. Journal of Big data, 3(1), 1-40.
32. Zheng C., Cham T., Cai J., 2019. Pluralistic Image Completion, arXiv preprint arXiv: 1903.04227.