Derin öğrenme tabanlı nesne algılama işlemlerinin farklı uygulama alanları

Abstract

Otomasyon, insan yaşamını ve çalışma koşullarını kolaylaştırmak için günlük yaşamda ve iş hayatında yaygınlaşmaktadır. Robotlar, sürücüsüz arabalar, insansız araçlar, robot kollar, otomatik fabrikalar vs. hayatımıza hızla girmektedir. Bu otomatikleştirilmiş aktörler için önemli görevlerden biri, çalışılacak ortamdaki nesneleri ve engelleri tanımaktır. Nesne algılama -nesnelerin cinsinin ve ortamdaki konumlarının belirlenmesi- bu görev için en önemli çözümlerden biridir. Evrişimli Sinir Ağı ve GPU işleme gibi derin öğrenme teknikleri ile nesne algılama işlemleri daha doğru ve hızlı sonuç üretmeye başlamış ve araştırmacıların dikkatini çekmiştir. Son yıllarda nesne algılama algoritmaları ve nesne algılamanın kullanımı ile ilgili birçok makale yayınlanmıştır. Nesne algılama algoritmaları hakkında inceleme makaleleri de bulunmaktadır, ancak genel itibariyle algoritmaları tanıtmış ve çok yaygın olarak bilinen uygulama alanlarına odaklanmışlardır. Diğer inceleme makalelerinden farklı olarak, bu çalışmada nesne algılama algoritmalarının çok geniş ve farklı uygulama alanına sahip olduğu gösterilmek istenmektedir. Çalışmada, derin öğrenmeye kısa bir giriş yapıldıktan sonra derin öğrenmeye dayalı standart nesne algılama algoritmaları ve bunların son yıllarda farklı araştırma alanlarındaki uygulamalarına yer verilerek gelecekteki çalışmalar için rehber olmak amaçlanmaktadır. Ayrıca makalelerde kullanılan veri setleri ve değerlendirme ölçütleri de listelenmiştir.

Keywords

• Nesne algılama
• Derin öğrenme
• CNN
• LSTM

DIFFERENT APPLICATION AREAS OF OBJECT DETECTION WITH DEEP LEARNING

Abstract

Automation is spread in all daily life and business activities to facilitate human life and working conditions. Robots, automated cars, unmanned vehicles, robot arms, automated factories etc. are getting place in our lives. For these automated actors, one important task is recognizing objects and obstacles in the target environment. Object detection, determining the objects and their location in the environment, is one of the most important solution for this task. With deep learning techniques like Convolutional Neural Network and GPU processing, object detection has become more accurate and faster, and getting attention of researchers. In recent years, many articles about object detection algorithms and usage of object detection have been published. There are surveys about the object detection algorithms, but they have introduced algorithms and focused on common application areas. With this survey, we aim to show that object detection algorithms have very large and different application area. In this study, we have given a brief introduction to deep learning. We have then focused on standard object detection algorithms based on deep learning and their applications in different research areas in recent years to give an idea for future works. Also, the datasets and evaluation metrics used in the research are listed.

Keywords

• Object detection
• deep learning
• CNN
• LSTM

References

1. Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016). End-to-end attention-based large vocabulary speech recognition. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4945–4949. https://doi.org/10.1109/ICASSP.2016.7472618
2. Caicedo, J. C., Goodman, A., Karhohs, K. W., Cimini, B. A., Ackerman, J., Haghighi, M., Heng, C. K., Becker, T., Doan, M., McQuin, C., Rohban, M., Singh, S., & Carpenter, A. E. (2019). Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl. Nature Methods. https://doi.org/10.1038/s41592-019-0612-7
3. Cevikalp, H., Benligiray, B., & Gerek, O. N. (2020). Semi-supervised robust deep neural networks for multi-label image classification. Pattern Recognition, 100, 107164. https://doi.org/https://doi.org/10.1016/j.patcog.2019.107164
4. Chen, J.-W., Lin, W.-J., Cheng, H.-J., Hung, C.-L., Lin, C.-Y., & Chen, S.-P. (2021). A Smartphone-Based Application for Scale Pest Detection Using Multiple-Object Detection Methods. Electronics, 10(4), 372. https://doi.org/10.3390/electronics10040372
5. Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., & Li, H. (2020). Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection. ArXiv:2012.15712.
6. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. https://doi.org/https://doi.org/10.1016/0364-0213(90)90002-E
7. Face Masks. (2020). https://www.kaggle.com/andrewmvd/face-mask-detection
8. Fan, D. P., Ji, G. P., Sun, G., Cheng, M. M., Shen, J., & Shao, L. (2020). Camouflaged object detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR42600.2020.00285

9. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3354–3361. https://doi.org/10.1109/CVPR.2012.6248074
10. Gers, F. A., Schmidhuber, J., & Cummins, F. (1999). Learning to forget: continual prediction with LSTM. 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), 2, 850–855 vol.2. https://doi.org/10.1049/cp:19991218
11. Girshick, R, Donahue, J., Darrell, T., & Malik, J. (2016). Region-Based Convolutional Networks for Accurate Object Detection and Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142–158. https://doi.org/10.1109/TPAMI.2015.2437384
12. Girshick, Ross. (2015). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV).
13. Hacıefendioğlu, K., Başağa, H. B., & Demir, G. (2021). Automatic detection of earthquake-induced ground failure effects through Faster R-CNN deep learning-based object detection using satellite images. Natural Hazards, 105(1), 383–403. https://doi.org/10.1007/s11069-020-04315-y
14. Han, F., Yao, J., Zhu, H., & Wang, C. (2020). Underwater Image Processing and Object Detection Based on Deep CNN Method. Journal of Sensors, 2020, 1–20. https://doi.org/10.1155/2020/6707328
15. Han, W., Zhang, Z., Caine, B., Yang, B., Sprunk, C., Alsharif, O., Ngiam, J., Vasudevan, V., Shlens, J., & Chen, Z. (2020). Streaming Object Detection for 3-D Point Clouds. http://arxiv.org/abs/2005.01864
16. Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation. https://doi.org/10.1162/neco.1997.9.8.1735
17. Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., & Igel, C. (2013). Detection of traffic signs in real-world images: The German traffic sign detection benchmark. The 2013 International Joint Conference on Neural Networks (IJCNN), 1–8. https://doi.org/10.1109/IJCNN.2013.6706807
18. Hung, J., Goodman, A., Ravel, D., Lopes, S. C. P., Rangel, G. W., Nery, O. A., Malleret, B., Nosten, F., Lacerda, M. V. G., Ferreira, M. U., Rénia, L., Duraisingh, M. T., Costa, F. T. M., Marti, M., & Carpenter, A. E. (2020). Keras R-CNN: library for cell detection in biological images using deep neural networks. BMC Bioinformatics, 21(1), 300. https://doi.org/10.1186/s12859-020-03635-x
19. Indolia, S., Goswami, A. K., Mishra, S. P., & Asopa, P. (2018). Conceptual Understanding of Convolutional Neural Network- A Deep Learning Approach. Procedia Computer Science. https://doi.org/10.1016/j.procs.2018.05.069
20. Jelodar, H., Wang, Y., Orji, R., & Huang, H. (2020). Deep Sentiment Classification and Topic Discovery on Novel Coronavirus or COVID-19 Online Discussions: NLP Using LSTM Recurrent Neural Network Approach. ArXiv:2004.11695.
21. Jing, R., Liu, S., Gong, Z., Wang, Z., Guan, H., Gautam, A., & Zhao, W. (2020). Object-based change detection for VHR remote sensing images based on a Trisiamese-LSTM. International Journal of Remote Sensing, 41(16), 6209–6231. https://doi.org/10.1080/01431161.2020.1734253
22. Kechaou, A., Martinez, M., Haurilet, M., & Stiefelhagen, R. (2020). Detective: An Attentive Recurrent Model for Sparse Object Detection. http://arxiv.org/abs/2004.12197
23. Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. http://arxiv.org/abs/1412.6980
24. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L. J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2017). Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journal of Computer Vision. https://doi.org/10.1007/s11263-016-0981-7
25. Kristo, M., Ivasic-Kos, M., & Pobar, M. (2020). Thermal Object Detection in Difficult Weather Conditions Using YOLO. IEEE Access, 8, 125459–125476. https://doi.org/10.1109/ACCESS.2020.3007481
26. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM, 60(6), 84–90. https://doi.org/10.1145/3065386
27. Law, H., & Deng, J. (2018). CornerNet: Detecting Objects as Paired Keypoints. Proceedings of the European Conference on Computer Vision (ECCV), 734–750.
28. Lawrence, S., Giles, C. L., & Fong, S. (2000). Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 12(1), 126–140. https://doi.org/10.1109/69.842255
29. Le Cun, Y., Jackel, L. D., Boser, B., Denker, J. S., Graf, H. P., Guyon, I., Henderson, D., Howard, R. E., & Hubbard, W. (1989). Handwritten digit recognition: applications of neural network chips and automatic learning. IEEE Communications Magazine, 27(11), 41–46. https://doi.org/10.1109/35.41400
30. Le, T.-N., Nguyen, T. V, Nie, Z., Tran, M.-T., & Sugimoto, A. (2019). Anabranch network for camouflaged object segmentation. Computer Vision and Image Understanding, 184, 45–56. https://doi.org/https://doi.org/10.1016/j.cviu.2019.04.006
31. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539 Lee, D.-H. (2020). CNN-based single object detection and tracking in videos and its application to drone detection. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-020-09924-0
32. Li, G., & Yu, Y. (2016). Visual Saliency Detection Based on Multiscale Deep CNN Features. IEEE Transactions on Image Processing, 25(11), 5012–5024. https://doi.org/10.1109/TIP.2016.2602079
33. Li, J., Zhao, R., Sun, E., Wong, J. H. M., Das, A., Meng, Z., & Gong, Y. (2020). High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7699–7703. https://doi.org/10.1109/ICASSP40776.2020.9054387
34. Li, R., Liu, H., Wang, X., & Qian, Y. (2018). DSBI: Double-Sided Braille Image Dataset and Algorithm Evaluation for Braille Dots Detection. Proceedings of the 2018 the 2nd International Conference on Video and Image Processing.
35. Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014). The Secrets of Salient Object Segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 280–287. https://doi.org/10.1109/CVPR.2014.43
36. Liu, Z., Li, Q., & Li, W. (2020). Deep layer guided network for salient object detection. Neurocomputing, 372, 55–63. https://doi.org/https://doi.org/10.1016/j.neucom.2019.09.018
37. Ljosa, V., Sokolnicki, K. L., & Carpenter, A. E. (2012). Annotated high-throughput microscopy image sets for validation. In Nature Methods. https://doi.org/10.1038/nmeth.2083
38. Loey, M., Manogaran, G., Taha, M. H. N., & Khalifa, N. E. M. (2020). Fighting against COVID-19: A novel deep learning model based on YOLO-v2 with ResNet-50 for medical face mask detection. Sustainable Cities and Society. https://doi.org/10.1016/j.scs.2020.102600
39. Medical Mask. (n.d.). Retrieved January 17, 2021, from https://humansintheloop.org/medical-mask-dataset/
40. Møgelmose, A., Trivedi, M. M., & Moeslund, T. B. (2012). Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2012.2209421
41. Ovodov, I. G. (2020). Optical Braille Recognition Using Object Detection CNN. ArXiv:2012.12412.
42. Öztemel, E. (2012). Yapay Sinir Ağları (3rd ed.). Papatya Yayıncılık Eğitim A.Ş.
43. Pascalvoc. (n.d.). Retrieved January 11, 2020, from http://host.robots.ox.ac.uk/pascal/VOC/
44. Pavan, G. S., Kumar, N., Karthik N., K., & Manikandan, J. (2020). Design of a Real-Time Speech Recognition System using CNN for Consumer Electronics. 2020 Zooming Innovation in Consumer Technologies Conference (ZINC), 5–10. https://doi.org/10.1109/ZINC50678.2020.9161432
45. Pi, Y., Nath, N. D., & Behzadan, A. H. (2020). Convolutional neural networks for object detection in aerial imagery for disaster response and recovery. Advanced Engineering Informatics, 43, 101009. https://doi.org/https://doi.org/10.1016/j.aei.2019.101009
46. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779–788.
47. Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
48. Sardoğan, M., Özen, Y., & Tuncer, A. (2020). Faster R-CNN Kullanarak Elma Yaprağı Hastalıklarının Tespiti. Düzce Üniversitesi Bilim ve Teknoloji Dergisi, 1110–1117. https://doi.org/10.29130/dubited.648387
49. Skurowski, P., Abdulameer, H., Błaszczyk, J., Depta, T., Kornacki, A., & Kozieł, P. (2017). CHAMELEON. http://kgwisc.aei.polsl.pl/index.php/en/dataset/63-animal-camouflage-analysis
50. Sorano, D., Carrara, F., Cintia, P., Falchi, F., & Pappalardo, L. (2020). Automatic Pass Annotation from Soccer VideoStreams Based on Object Detection and LSTM. http://arxiv.org/abs/2007.06475
51. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., … Anguelov, D. (2020). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2443–2451. https://doi.org/10.1109/CVPR42600.2020.00252
52. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.308
53. Tassinari, P., Bovo, M., Benni, S., Franzoni, S., Poggi, M., Mammi, L. M. E., Mattoccia, S., Di Stefano, L., Bonora, F., Barbaresi, A., Santolini, E., & Torreggiani, D. (2021). A computer vision approach based on deep learning for the detection of dairy cows in free stall barn. Computers and Electronics in Agriculture, 182, 106030. https://doi.org/10.1016/j.compag.2021.106030
54. Tsai, W.-J., Huang, Z.-J., & Chung, C.-E. (2020). Joint Detection, Re-Identification, And Lstm In Multi-Object Tracking. 2020 IEEE International Conference on Multimedia and Expo (ICME), 1–6. https://doi.org/10.1109/ICME46284.2020.9102884
55. Turan, S., & Bilgin, G. (2019). Semantic nuclei segmentation with deep learning on breast pathology images. 2019 Scientific Meeting on Electrical-Electronics Biomedical Engineering and Computer Science (EBBT), 1–4. https://doi.org/10.1109/EBBT.2019.8741715
56. Ufldl Tutorial. (n.d.). UFLDL Tutorial. Retrieved December 24, 2020, from http://ufldl.stanford.edu/tutorial/
57. Vinayakumar, R., Soman, K. P., & Poornachandran, P. (2017). Applying convolutional neural network for network intrusion detection. 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1222–1228. https://doi.org/10.1109/ICACCI.2017.8126009
58. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., & Ruan, X. (2017). Learning to Detect Salient Objects with Image-Level Supervision. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3796–3805. https://doi.org/10.1109/CVPR.2017.404
59. Wu, A., Zhang, Q., Fang, W., Deng, H., Jiang, S., & Liu, Q. (2020). Mask R-CNN Based Object Detection for Intelligent Wireless Power Transfer. http://arxiv.org/abs/2004.10021
60. Xiao, H., Xu, J., & Shi, J. (2020). Exploring diverse and fine-grained caption for video by incorporating convolutional architecture into LSTM-based model. Pattern Recognition Letters, 129, 173–180. https://doi.org/https://doi.org/10.1016/j.patrec.2019.11.003
61. Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical Saliency Detection. CVPR 2013.
62. Yang, C., Zhang, L., Lu, H., Ruan, X., & Yang, M.-H. (2013). Saliency detection via graph-based manifold ranking. Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference On, 3166–3173.
63. Yuan, D., Li, X., He, Z., Liu, Q., & Lu, S. (2020). Visual object tracking with adaptive structural convolutional network.
64. Knowledge-Based Systems, 194, 105554. https://doi.org/https://doi.org/10.1016/j.knosys.2020.105554 Zhang, J., Huang, M., Jin, X., & Li, X. (2017). A real-time Chinese traffic sign detection algorithm based on modified YOLOv2. Algorithms. https://doi.org/10.3390/a10040127
65. Zhang, J., Xie, Z., Sun, J., Zou, X., & Wang, J. (2020). A Cascaded R-CNN with Multiscale Attention and Imbalanced Samples for Traffic Sign Detection. IEEE Access. https://doi.org/10.1109/ACCESS.2020.2972338
66. Zhang, W., Xu, L., Li, Z., Lu, Q., & Liu, Y. (2016). A Deep-Intelligence Framework for Online Video Processing. IEEE Software, 33(2), 44–51. https://doi.org/10.1109/MS.2016.31
67. Zhao, D., Chang, Z., & Guo, S. (2020). Cross-scale fusion detection with global attribute for dense captioning. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.09.055
68. Zhu, J., Guo, Y., Yue, F., Yuan, H., Yang, A., Wang, X., & Rong, M. (2020). A deep learning method to detect foreign objects for inspecting power transmission lines. IEEE Access. https://doi.org/10.1109/ACCESS.2020.2995608