Semantic Based Image Retrieval-A Survey

Abstract

As a result of rapid technological development, operating with massive data has become a common situation. There is a need for machine learning to process these data and extract meaningful information, and make a decision from them. Current studies related to identifying objects from the image are driven to Semantic- Based Image Retrieval. The studies done in this field aim to dismiss the discrepancies among the low-level color, shape, texture characteristics and picture recognition by people that are extracted from images by machines known as the Semantic Gap, that are signified as high-level concepts. Therefore, definite ontologies are created to determine characteristics of the concept of a particular domain and show the relationship between them by advancing the research on this area. Through ontologies, information is transformed into a structure so computers can process and create a meaningful relationship between information. In this study, a compilation on Semantic-Based Image Retrieval – SBIR is done. SBIR aims to overcome the bottleneck faced in the search operations created by Content-Based Image Retrieval (CBIR) and shown as a Semantic Gap. In the studies done, significant progress in problem-solving through the use of the Ontology concept is observed.

Keywords

• Image Retrieval
• Semantic Gap
• Deep Learning
• Semantic Based Image Retrieval

Anlamsal Tabanlı Görüntü Erişimi Üzerine Bir Derleme

Abstract

Bilgisayar teknolojisinin hızlı gelişmesi sonucunda çok büyük miktarlarda verilerle çalışmak olağan bir durum haline gelmiştir. Bu verilerin işlenmesi, verilerden anlamlı bilgiler çıkarılması ve kararlar alınması için makine öğrenmesine ihtiyaç duyulmaktadır. Görüntü içerisinden nesnelerin algılanmasına yönelik son zamanlarda yapılan çalışmalar özellikle Anlamsal Tabanlı Görüntü Erişimi alanına doğru yönelmektedir. Bu alanda yapılan çalışmalar ile Anlamsal Boşluk olarak adlandırılan ve görüntülerden makineler tarafından çıkarılan düşük düzeydeki renk, şekil, doku (color, shape, texture) özellikleri ile insanlar tarafından resimlerden algılanan ve yüksek düzey olarak ifade edilen kavramlar arasındaki uyuşmazlıkların giderilmesine yoğunlaşmaktadır. Bu amaçla, belirli bir bilgi alanına (domain) ait kavramların özelliklerini ve aralarındaki ilişkileri göstermek için iyi tanımlanmış ontolojiler oluşturulmakta ve arama işlemi bu yönde ilerlemektedir. Ontolojiler kullanılarak bilgiler bilgisayarların işleyebileceği biçime dönüştürülmekte ve bilgiler arasında anlamlı ilişkiler oluşturulabilmektedir. Bu çalışmada Anlamsal Tabanlı Görüntü Erişimi (Semantic Based Image Retrieval - SBIR) üzerine bir derleme yapılmıştır. SBIR ile amaç İçerik Tabanlı Görüntü Erişimi (Content Based Image Retrieval - CBIR) ile yapılan arama işlemlerinde karşılaşılan ve Anlamsal Boşluk (Semantic Gap) olarak ifade edilen darboğazın aşılmasıdır. Yapılan çalışmalarda Ontoloji (Ontology) kavramının kullanılmasıyla problemin çözümünde önemli bir gelişme yaşandığı gözlemlenmiştir.

Keywords

• Görüntü Alma
• Anlamsal Boşluk
• Derin Öğrenme
• Semantik Görüntü Alma

References

1. Alkhawlani, M., Elmogy, M., & El Bakry, H. (2015). Text-based, content-based, and semantic-based image retrievals: A survey. In International Journal of Computer and Information Technology (ISSN: 2279–0764) (Vol. 4, Issue 01).
2. Alpaydın, E. (2004). Introduction to machine learning. MIT Press.
3. Alzu’bi, A., Amira, A., & Ramzan, N. (2015). Semantic content-based image retrieval: A comprehensive study. Journal of Visual Communication and Image Representation, 32, 20–54. https://doi.org/10.1016/j.jvcir.2015.07.012
4. Alzu’bi, A., Amira, A., & Ramzan, N. (2017). Content-based image retrieval with compact deep convolutional features. In Neurocomputing (Vol. 249, pp. 95–105). https://doi.org/10.1016/j.neucom.2017.03.072
5. Ashraf, R., Ahmed, M., Jabbar, S., Khalid, S., Ahmad, A., Din, S., & Jeon, G. (2018). Content Based Image Retrieval by Using Color Descriptor and Discrete Wavelet Transform. Journal of Medical Systems, 42(3). https://doi.org/10.1007/s10916-017-0880-7
6. Aslandogan, Y. A., & Yu, C. T. (1999). Techniques and systems for image and video retrieval. In IEEE Transactions on Knowledge and Data Engineering (Vol. 11, Issue 1, pp. 56–63). https://doi.org/10.1109/69.755615
7. Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615
8. Bouchakwa, M., Ayadi, Y., & Amous, I. (2020). Multi-level diversification approach of semantic-based image retrieval results. Progress in Artificial Intelligence, 9(1), 1–30. https://doi.org/10.1007/s13748-019-00195-x

9. Chen, H., Guo, A. Bin, Ni, W., & Cheng, Y. (2020). Improving the representation of image descriptions for semantic image retrieval with RDF. Journal of Visual Communication and Image Representation, 73(August 2019), 102934. https://doi.org/10.1016/j.jvcir.2020.102934
10. De Geus, D., Meletis, P., & Dubbelman, G. (2020). Fast panoptic segmentation network. IEEE Robotics and Automation Letters, 5(2), 1742–1749. https://doi.org/10.1109/LRA.2020.2969919
11. Deserno, T. M., Antani, S., & Long, R. (2009). Ontology of gaps in content-based image retrieval. In Journal of Digital Imaging (Vol. 22, Issue 2, pp. 202– 215). https://doi.org/10.1007/s10278-007-9092-x
12. Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016). Deep learning for visual understanding: A review. In Neurocomputing (Vol. 187, pp. 27–48). https://doi.org/10.1016/j.neucom.2015.09.116
13. Li, Y., Wang, Y., & Huang, X. (2007). A relation-based search engine in Semantic Web. IEEE Transactions on Knowledge and Data Engineering, 19(2), 273–281. https://doi.org/10.1109/TKDE.2007.18
14. Liu, Y., Zhang, D., Lu, G., & Ma, W. Y. (2007). A survey of content-based image retrieval with high-level semantics. Pattern Recognition, 40(1), 262–282. https://doi.org/10.1016/j.patcog.2006.04.045
15. Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. In IEEE. https://doi.org/10.1109/CVPR.2015.7298965
16. Ma, H., Zhu, J., Lyu, M. R. T., & King, I. (2010). Bridging the semantic gap between image contents and tags. IEEE Transactions on Multimedia, 12(5), 462–473. https://doi.org/10.1109/TMM.2010.2051360
17. Mezaris, V., Kompatsiaris, I., & Strintzis, M. G. (2003). An ontology approach to object-based image retrieval. Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429), 3, II-511–514. https://doi.org/10.1109/ICIP.2003.1246729
18. Minu, R. I., & Thyagharajan, K. K. (2014). Semantic rule based image visual feature ontology creation. International Journal of Automation and Computing, 11(5), 489–499. https://doi.org/10.1007/s11633-014-0832-3
19. Ngo, T. G., Ngo, Q. T., & Nguyen, D. D. (2016). Image Retrieval with relevance feedback using SVM active learning. International Journal of Electrical and Computer Engineering, 6(6), 3238–3246. https://doi.org/10.11591/ijece.v6i6.11631
20. Noh, H., Hong, S., & Han, B. (2015). Learning Deconvolution Network for Semantic Segmentation (Vol. 1). https://doi.org/10.1109/ICCV.2015.178
21. Pang, Y., Li, Y., Shen, J., & Shao, L. (2019). Towards bridging semantic gap to improve semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob(Iccv), 4229–4238. https://doi.org/10.1109/ICCV.2019.00433
22. Parsons, S. (2009). A Semantic Web Primer, Second Edition by Antoniou Grigoris and Harmelen Frank van, MIT Press, 288 pp.. In The Knowledge Engineering Review (Vol. 24, Issue 4). https://doi.org/10.1017/s0269888909990117
23. Rizwan I Haque, I., & Neubert, J. (2020). Deep learning approaches to biomedical image segmentation. In Informatics in Medicine Unlocked (Vol. 18). https://doi.org/10.1016/j.imu.2020.100297
24. Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000). Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1349–1380. https://doi.org/10.1109/34.895972
25. Song, K., Li, F., Long, F., Wang, J., & Ling, Q. (2018). Discriminative Deep Feature Learning for Semantic-Based Image Retrieval. IEEE Access, 6, 44268– 44280. https://doi.org/10.1109/ACCESS.2018.2862464
26. Tzelepi, M., & Tefas, A. (2018). Deep convolutional learning for Content Based Image Retrieval. In Neurocomputing (Vol. 275, pp. 2467–2478). https://doi.org/10.1016/j.neucom.2017.11.022
27. Wang, Q., Lai, J., Claesen, L., Yang, Z., Lei, L., & Liu, W. (2020). A novel feature representation: Aggregating convolution kernels for image retrieval. Neural Networks, 130, 1–10. https://doi.org/10.1016/j.neunet.2020.06.010
28. Wu, Q. (2020). Image retrieval method based on deep learning semantic feature extraction and regularization softmax. Multimedia Tools and Applications, 79(13–14), 9419–9433. https://doi.org/10.1007/s11042-019-7605-5
29. Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2020). Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing, 104042. https://doi.org/https://doi.org/10.1016/j.imavis.2020.104042
30. Zhao, R., & Grosky, W. I. (2002). Narrowing the semantic gap - Improved text-based web document retrieval using visual features. IEEE Transactions on Multimedia, 4(2), 189–200. https://doi.org/10.1109/TMM.2002.1017733
31. Zhu, H. (2020). Massive-scale image retrieval based on deep visual feature representation. Journal of Visual Communication and Image Representation, 70. https://doi.org/10.1016/j.jvcir.2019.102738
32. WordNet. Retrieved from https://wordnet.princeton.edu, (22.11.2020)