A Review on Using Deep Learning Techniques to Bridge The Semantic Gap

Abstract

The concept of semantic gap arises from the differences between the basic attributes such as color, texture and shape extracted from the data by machines and the conceptual results defined by human perception when looking at the same data. Various ways and methods have been tried in the literature to bridge this gap. In these studies, it is seen that ontological systems have been developed and search operations have been focused in this direction. The main purpose of our study is to present the studies on Semantic Based Image Retrieval in general and the studies on overcoming the Semantic Gap problem that arises in the search for Content Based Image Retrieval. We present a concise and systematic comparative literature review by analyzing various international journal and conference publications in the literature. Scientific discussion is also included.

Keywords

• Artificial Intelligence
• Image Processing
• Semantic Gap
• Deep Learning
• Convolutional Neural Network

Anlamsal Boşluk Doldurulmasında Derin Öğrenme Yöntemlerinin Kullanılması Üzerine Bir İnceleme

Abstract

Anlamsal boşluk kavramı, makineler aracılığıyla veriden elde edilen temel renk, doku ve şekil gibi özniteliklerle insan tarafından aynı veriye bakıldığında algı yoluyla tanımlanan kavramsal sonuçların farklılıklarından doğmaktadır. Bu boşluğun giderilmesi için çeşitli yol ve yöntemler literatürdeki çalışmalarda denenmiştir. Bu çalışmalarda, ontolojik sistemlerin geliştirildiği ve arama işlemlerinin de bu tarafa yoğunlaştırılmasının sağlandığı görülmektedir. Çalışmamızın ana amacı genel olarak Anlamsal Tabanlı Görüntüden Bilgi Elde Etme çalışmalarıyla İçerik Tabanlı Görüntüden Bilgi Elde Etme için gerçekleştirilen aramada ortaya çıkan Anlamsal Boşluk sorunun üstesinden gelinmesine dair yapılan çalışmaları ortaya koymaktır. Çalışmamızda literatürdeki çeşitli uluslararası dergi ve konferans yayınları incelenerek, kıyaslamalı özlü ve sistematik bir literatür taraması sunulmaktadır. Bilimsel tartışmaya da yer verilmektedir.

Keywords

• Yapay Zekâ
• Görüntü İşleme
• Anlamsal Boşluk
• Derin Öğrenme
• Evrişimli Sinir Ağı

References

1. [1] Alpaydın, E. (2004). Introduction to machine learning. MIT Press.
2. [2] Aslandogan, Y. A., & Yu, C. T. (1999). Techniques and systems for image and video retrieval. In IEEE Transactions on Knowledge and Data Engineering (Vol. 11, Issue 1, pp. 56–63). https://doi.org/10.1109/69.755615
3. [3] Alkhawlani, M., Elmogy, M., & El Bakry, H. (2015). Text-based, content-based, and semantic-based image retrievals: A survey. In International Journal of Computer and Information Technology (ISSN: 2279–0764) (Vol. 4, Issue 01).
4. [4] Ngo, T. G., Ngo, Q. T., & Nguyen, D. D. (2016). Image Retrieval with relevance feedback using SVM active learning. International Journal of Electrical and Computer Engineering, 6(6), 3238–3246. https://doi.org/10.11591/ijece.v6i6.11631
5. [5] Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. In IEEE. https://doi.org/10.1109/CVPR.2015.7298965
6. [6] El Shaer, Mohamed & Wisdom, Scott & Mishra, Taniya. (2019). Transfer Learning From Sound Representations For Anger Detection in Speech. Erişim Adresi: https://www.researchgate.net/publication/330923908_Transfer_Learning_From_Sound_Representations_For_Anger_Detection_in_Speech
7. [7] Tzelepi, M., & Tefas, A. (2018). Deep convolutional learning for Content Based Image Retrieval. In Neurocomputing (Vol. 275, pp. 2467–2478). https://doi.org/10.1016/j.neucom.2017.11.022
8. [8] Oxford 5K Veri Kümesi. Erişim Adresi (23.06.2023): https://www.robots.ox.ac.uk/~vgg/data/oxbuildings/

9. [9] Alzu’bi, A., Amira, A., & Ramzan, N. (2017). Content-based image retrieval with compact deep convolutional features. In Neurocomputing (Vol. 249, pp. 95–105). https://doi.org/10.1016/j.neucom.2017.03.072
10. [10] Noh, H., Hong, S., & Han, B. (2015). Learning Deconvolution Network for Semantic Segmentation (Vol. 1). In: ICCV.. https://doi.org/10.1109/ICCV.2015.178
11. [11] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615
12. [12] De Geus, D., Meletis, P., & Dubbelman, G. (2020). Fast panoptic segmentation network. IEEE Robotics and Automation Letters, 5(2), 1742–1749. https://doi.org/10.1109/LRA.2020.2969919
13. [13] Suhasini, P. S., Krishna, K. S. & Krishna, I. V. (2008). Graph Based Segmentation in Content Based Image Retrieval. Journal of Computer Science, 4(8), 699-705. https://doi.org/10.3844/jcssp.2008.699.705
14. [14] Ozden M., Polat E. (2007) A color image segmentation approach for content-based image retrieval. Pattern Recognition 40(4):1318–1325. https://doi.org/10.1016/j.patcog.2006.08.013
15. [15] Rizwan I Haque, I., & Neubert, J. (2020). Deep learning approaches to biomedical image segmentation. In Informatics in Medicine Unlocked (Vol. 18). https://doi.org/10.1016/j.imu.2020.100297
16. [16] Ma, H., Zhu, J., Lyu, M. R. T., & King, I. (2010). Bridging the semantic gap between image contents and tags. IEEE Transactions on Multimedia, 12(5), 462–473. https://doi.org/10.1109/TMM.2010.2051360
17. [17] Pang, Y., Li, Y., Shen, J., & Shao, L. (2019). Towards bridging semantic gap to improve semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob(Iccv), 4229–4238. https://doi.org/10.1109/ICCV.2019.00433
18. [18] Philbin, J., Chum, O., Isard, M., & Sivic, J., (2008). Lost in quantization: Improving particular object retrieval in large scale image databases. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR. 1–8.
19. [19] Zhao, R., & Grosky, W. I. (2002). Narrowing the semantic gap - Improved text-based web document retrieval using visual features. IEEE Transactions on Multimedia, 4(2), 189–200. https://doi.org/10.1109/TMM.2002.1017733
20. [20] Wu, L., Hua, X.-S., Yu, N., Ma W.-Y., & Li, S. (2008). Flickr distance, In: Proc. MM'08, 31-40, WordNet Veri Kümesi. Erişim Adresi (23.06.2023): https://wordnet.princeton.edu
21. [21] Auchard, E.. (2007). Flickr to map the world's latest photo hotspots, Reuters, CNet. Erişim Adresi (23.06.2023): https://www.cnet.com/tech/tech-industry/flickr-to-map-the-worlds-latest-photo-hot-spots/
22. [22] Everingham, M., Eslami, S. M. A. Van Gool, L. J., Williams, C. K. I. Winn J. M. & Zisserman, A. (2015). The pascal visual object classes challenge: A retrospective, IJCV, 2015.
23. [23] Brostow, G. J. Shotton, J. Fauqueur J. & Cipolla, R. (2008). Segmentation and recognition using structure from motion point clouds, In: Proc. ECCV.
24. [24] Scopus Web Sitesi. Erişim Adresi (23.06.2023): https://www.scopus.com/home.uri
25. [25] Web of Science Web Sitesi. Erişim Adresi (23.06.2023): https://access.clarivate.com/
26. [26] IEEE Web Sitesi. Erişim Adresi (23.06.2023): https://ieeexplore.ieee.org/Xplore/home.jsp
27. [27] WordNet Veri Kümesi. Erişim Adresi (23.06.2023): https://wordnet.princeton.edu
28. [28] Ashraf, R., Ahmed, M., Jabbar, S., Khalid, S., Ahmad, A., Din, S., & Jeon, G. (2018). Content Based Image Retrieval by Using Color Descriptor and Discrete Wavelet Transform. Journal of Medical Systems, 42(3). https://doi.org/10.1007/s10916-017-0880-7
29. [29] Yenigalla S.C., Rao K.S., Ngangbam P.S.. (2023) Implementation of Content-Based Image Retrieval Using Artificial Neural Networks. Engineering Proceedings; 34(1):25. https://doi.org/10.3390/HMAM2-14161
30. [30] Sikandar S., Mahum R., Alsalman A. (2023) A Novel Hybrid Approach for a Content-Based Image Retrieval Using Feature Fusion. Applied Sciences; 13(7):4581. https://doi.org/10.3390/app13074581
31. [31] Song, K., Li, F., Long, F., Wang, J., & Ling, Q. (2018). Discriminative Deep Feature Learning for Semantic-Based Image Retrieval. IEEE Access, 6, 44268– 44280. https://doi.org/10.1109/ACCESS.2018.2862464
32. [32] Wu, Q. (2020). Image retrieval method based on deep learning semantic feature extraction and regularization softmax. Multimedia Tools and Applications, 79(13–14), 9419–9433. https://doi.org/10.1007/s11042-019-7605-5
33. [33] Bouchakwa, M., Ayadi, Y., & Amous, I. (2020). Multi-level diversification approach of semantic-based image retrieval results. Progress in Artificial Intelligence, 9(1), 1–30. https://doi.org/10.1007/s13748-019-00195-x
34. [34] Li, Y., Wang, Y., & Huang, X. (2007). A relation-based search engine in Semantic Web. IEEE Transactions on Knowledge and Data Engineering, 19(2), 273–281. https://doi.org/10.1109/TKDE.2007.18
35. [35] Minu, R. I., & Thyagharajan, K. K. (2014). Semantic rule based image visual feature ontology creation. International Journal of Automation and Computing, 11(5), 489–499. https://doi.org/10.1007/s11633-014-0832-3
36. [36] Gasi, A., Ensari, T. ve Dagtekin, M. (2021). Anlamsal tabanlı görüntü erişimi üzerine bir derleme. Acta Infologica, 5(2), 445-457. https://doi.org/10.26650/acin.835241