Semantic Based Image Retrieval-A Survey

Yıl 2021, Cilt: 5 Sayı: 2, 445 - 457, 30.12.2021

Akif Gaşi Tolga Ensari Mustafa Dağtekin

Öz

As a result of rapid technological development, operating with massive data has become a common situation. There is a need for machine learning to process these data and extract meaningful information, and make a decision from them. Current studies related to identifying objects from the image are driven to Semantic- Based Image Retrieval. The studies done in this field aim to dismiss the discrepancies among the low-level color, shape, texture characteristics and picture recognition by people that are extracted from images by machines known as the Semantic Gap, that are signified as high-level concepts. Therefore, definite ontologies are created to determine characteristics of the concept of a particular domain and show the relationship between them by advancing the research on this area. Through ontologies, information is transformed into a structure so computers can process and create a meaningful relationship between information. In this study, a compilation on Semantic-Based Image Retrieval – SBIR is done. SBIR aims to overcome the bottleneck faced in the search operations created by Content-Based Image Retrieval (CBIR) and shown as a Semantic Gap. In the studies done, significant progress in problem-solving through the use of the Ontology concept is observed.

Anahtar Kelimeler

Image Retrieval, Semantic Gap, Deep Learning, Semantic Based Image Retrieval

Anlamsal Tabanlı Görüntü Erişimi Üzerine Bir Derleme

Öz

Bilgisayar teknolojisinin hızlı gelişmesi sonucunda çok büyük miktarlarda verilerle çalışmak olağan bir durum haline gelmiştir. Bu verilerin işlenmesi, verilerden anlamlı bilgiler çıkarılması ve kararlar alınması için makine öğrenmesine ihtiyaç duyulmaktadır. Görüntü içerisinden nesnelerin algılanmasına yönelik son zamanlarda yapılan çalışmalar özellikle Anlamsal Tabanlı Görüntü Erişimi alanına doğru yönelmektedir. Bu alanda yapılan çalışmalar ile Anlamsal Boşluk olarak adlandırılan ve görüntülerden makineler tarafından çıkarılan düşük düzeydeki renk, şekil, doku (color, shape, texture) özellikleri ile insanlar tarafından resimlerden algılanan ve yüksek düzey olarak ifade edilen kavramlar arasındaki uyuşmazlıkların giderilmesine yoğunlaşmaktadır. Bu amaçla, belirli bir bilgi alanına (domain) ait kavramların özelliklerini ve aralarındaki ilişkileri göstermek için iyi tanımlanmış ontolojiler oluşturulmakta ve arama işlemi bu yönde ilerlemektedir. Ontolojiler kullanılarak bilgiler bilgisayarların işleyebileceği biçime dönüştürülmekte ve bilgiler arasında anlamlı ilişkiler oluşturulabilmektedir. Bu çalışmada Anlamsal Tabanlı Görüntü Erişimi (Semantic Based Image Retrieval - SBIR) üzerine bir derleme yapılmıştır. SBIR ile amaç İçerik Tabanlı Görüntü Erişimi (Content Based Image Retrieval - CBIR) ile yapılan arama işlemlerinde karşılaşılan ve Anlamsal Boşluk (Semantic Gap) olarak ifade edilen darboğazın aşılmasıdır. Yapılan çalışmalarda Ontoloji (Ontology) kavramının kullanılmasıyla problemin çözümünde önemli bir gelişme yaşandığı gözlemlenmiştir.

Anahtar Kelimeler

Görüntü Alma, Anlamsal Boşluk, Derin Öğrenme, Semantik Görüntü Alma

Kaynakça

• Alkhawlani, M., Elmogy, M., & El Bakry, H. (2015). Text-based, content-based, and semantic-based image retrievals: A survey. In International Journal of Computer and Information Technology (ISSN: 2279–0764) (Vol. 4, Issue 01).
• Alpaydın, E. (2004). Introduction to machine learning. MIT Press.
• Alzu’bi, A., Amira, A., & Ramzan, N. (2015). Semantic content-based image retrieval: A comprehensive study. Journal of Visual Communication and Image Representation, 32, 20–54. https://doi.org/10.1016/j.jvcir.2015.07.012
• Alzu’bi, A., Amira, A., & Ramzan, N. (2017). Content-based image retrieval with compact deep convolutional features. In Neurocomputing (Vol. 249, pp. 95–105). https://doi.org/10.1016/j.neucom.2017.03.072
• Ashraf, R., Ahmed, M., Jabbar, S., Khalid, S., Ahmad, A., Din, S., & Jeon, G. (2018). Content Based Image Retrieval by Using Color Descriptor and Discrete Wavelet Transform. Journal of Medical Systems, 42(3). https://doi.org/10.1007/s10916-017-0880-7
• Aslandogan, Y. A., & Yu, C. T. (1999). Techniques and systems for image and video retrieval. In IEEE Transactions on Knowledge and Data Engineering (Vol. 11, Issue 1, pp. 56–63). https://doi.org/10.1109/69.755615
• Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615
• Bouchakwa, M., Ayadi, Y., & Amous, I. (2020). Multi-level diversification approach of semantic-based image retrieval results. Progress in Artificial Intelligence, 9(1), 1–30. https://doi.org/10.1007/s13748-019-00195-x
• Chen, H., Guo, A. Bin, Ni, W., & Cheng, Y. (2020). Improving the representation of image descriptions for semantic image retrieval with RDF. Journal of Visual Communication and Image Representation, 73(August 2019), 102934. https://doi.org/10.1016/j.jvcir.2020.102934
• De Geus, D., Meletis, P., & Dubbelman, G. (2020). Fast panoptic segmentation network. IEEE Robotics and Automation Letters, 5(2), 1742–1749. https://doi.org/10.1109/LRA.2020.2969919
• Deserno, T. M., Antani, S., & Long, R. (2009). Ontology of gaps in content-based image retrieval. In Journal of Digital Imaging (Vol. 22, Issue 2, pp. 202– 215). https://doi.org/10.1007/s10278-007-9092-x
• Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016). Deep learning for visual understanding: A review. In Neurocomputing (Vol. 187, pp. 27–48). https://doi.org/10.1016/j.neucom.2015.09.116
• Li, Y., Wang, Y., & Huang, X. (2007). A relation-based search engine in Semantic Web. IEEE Transactions on Knowledge and Data Engineering, 19(2), 273–281. https://doi.org/10.1109/TKDE.2007.18
• Liu, Y., Zhang, D., Lu, G., & Ma, W. Y. (2007). A survey of content-based image retrieval with high-level semantics. Pattern Recognition, 40(1), 262–282. https://doi.org/10.1016/j.patcog.2006.04.045
• Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. In IEEE. https://doi.org/10.1109/CVPR.2015.7298965
• Ma, H., Zhu, J., Lyu, M. R. T., & King, I. (2010). Bridging the semantic gap between image contents and tags. IEEE Transactions on Multimedia, 12(5), 462–473. https://doi.org/10.1109/TMM.2010.2051360
• Mezaris, V., Kompatsiaris, I., & Strintzis, M. G. (2003). An ontology approach to object-based image retrieval. Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429), 3, II-511–514. https://doi.org/10.1109/ICIP.2003.1246729
• Minu, R. I., & Thyagharajan, K. K. (2014). Semantic rule based image visual feature ontology creation. International Journal of Automation and Computing, 11(5), 489–499. https://doi.org/10.1007/s11633-014-0832-3
• Ngo, T. G., Ngo, Q. T., & Nguyen, D. D. (2016). Image Retrieval with relevance feedback using SVM active learning. International Journal of Electrical and Computer Engineering, 6(6), 3238–3246. https://doi.org/10.11591/ijece.v6i6.11631
• Noh, H., Hong, S., & Han, B. (2015). Learning Deconvolution Network for Semantic Segmentation (Vol. 1). https://doi.org/10.1109/ICCV.2015.178
• Pang, Y., Li, Y., Shen, J., & Shao, L. (2019). Towards bridging semantic gap to improve semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob(Iccv), 4229–4238. https://doi.org/10.1109/ICCV.2019.00433
• Parsons, S. (2009). A Semantic Web Primer, Second Edition by Antoniou Grigoris and Harmelen Frank van, MIT Press, 288 pp.. In The Knowledge Engineering Review (Vol. 24, Issue 4). https://doi.org/10.1017/s0269888909990117
• Rizwan I Haque, I., & Neubert, J. (2020). Deep learning approaches to biomedical image segmentation. In Informatics in Medicine Unlocked (Vol. 18). https://doi.org/10.1016/j.imu.2020.100297
• Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000). Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1349–1380. https://doi.org/10.1109/34.895972
• Song, K., Li, F., Long, F., Wang, J., & Ling, Q. (2018). Discriminative Deep Feature Learning for Semantic-Based Image Retrieval. IEEE Access, 6, 44268– 44280. https://doi.org/10.1109/ACCESS.2018.2862464
• Tzelepi, M., & Tefas, A. (2018). Deep convolutional learning for Content Based Image Retrieval. In Neurocomputing (Vol. 275, pp. 2467–2478). https://doi.org/10.1016/j.neucom.2017.11.022
• Wang, Q., Lai, J., Claesen, L., Yang, Z., Lei, L., & Liu, W. (2020). A novel feature representation: Aggregating convolution kernels for image retrieval. Neural Networks, 130, 1–10. https://doi.org/10.1016/j.neunet.2020.06.010
• Wu, Q. (2020). Image retrieval method based on deep learning semantic feature extraction and regularization softmax. Multimedia Tools and Applications, 79(13–14), 9419–9433. https://doi.org/10.1007/s11042-019-7605-5
• Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2020). Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing, 104042. https://doi.org/https://doi.org/10.1016/j.imavis.2020.104042
• Zhao, R., & Grosky, W. I. (2002). Narrowing the semantic gap - Improved text-based web document retrieval using visual features. IEEE Transactions on Multimedia, 4(2), 189–200. https://doi.org/10.1109/TMM.2002.1017733
• Zhu, H. (2020). Massive-scale image retrieval based on deep visual feature representation. Journal of Visual Communication and Image Representation, 70. https://doi.org/10.1016/j.jvcir.2019.102738
• WordNet. Retrieved from https://wordnet.princeton.edu, (22.11.2020)