DDL: A new deep learning based approach for multiple house numbers detection and clustering

Öz

Character recognition in natural images is a very difficult problem due to many factors such as variability of light, background clutter, severe blur, inconsistent resolution and different scale depth. In addition to these features, distortions in characters and numbers are encountered in street view photographs with the effect of natural events. Detecting and reading house numbers from street views is a computer vision problem that falls under the category of natural scene text recognition. Convolutional neural network (CNN) model is one of the most commonly used deep learning (DL) methods in image analysis. In this study, firstly, CNN based DL method was applied to read characters from pictures that contain house numbers in their natural image. However, successful results could not be obtained, especially in cases where there are more than one house number in the image or when the depths are very variable. A new approach DDL (deep in deep learning) using two different CNN models was proposed to increase the accuracy of the DL method and also to reduce the data size created by natural images. The performance of the proposed DDL approach was compared with the performance of the DL approach using real data consisting of 17,618 images with 113 GB (gigabyte) size consisting of building street images with GPS location information taken from 35 neighborhoods of Kayseri Metropolitan Municipality (KBB) Yeşilhisar district for 2019. Experimental results showed that the proposed DDL approach produced more accurate results and used less storage space than DL approach

Anahtar Kelimeler

• Convolutional neural network
• Deep learning
• Multiple house numbers detection and clustering

DDL: Çoklu kapı numarası tespit etme ve kümeleme için derin öğrenme tabanlı yeni bir yaklaşım

Öz

Işığın değişkenliği, arka plan karmaşası, şiddetli bulanıklık, tutarsız çözünürlük ve farklı ölçekli derinlik gibi birçok faktörden dolayı doğal görüntülerde karakter tanıma oldukça zor problemdir. Bu özelliklerin yanı sıra sokak görünüm fotoğraflarında doğa olaylarının da etkisiyle karakterlerde ve sayılarda bozulmalara rastlanır. Sokak görünümlerinden kapı numaralarını tespit etmek ve okumak, doğal sahne metni tanıma kategorisine giren bir bilgisayar görme problemidir. Evrişimsel sinir ağı (convolutional neural network - CNN) modeli görüntü analizlerinde en sık kullanılan derin öğrenme (deep learning - DL) yöntemlerinden biridir. Bu çalışmada, ilk olarak doğal görüntülerinde kapı numarası bulunan görüntülerden karakter okumak için CNN temelli DL yöntemi uygulanmıştır. Ancak, özellikle görüntüde birden fazla kapı numarasının olduğu veya derinliklerin çok değişken olduğu durumlarda yeterince başarılı sonuçlar elde edilememiştir. DL yönteminin doğruluğunu artırmak aynı zamanda doğal görüntülerin oluşturduğu veri boyutunu azaltmak için farklı iki adet CNN modeli kullanan yeni bir yaklaşım DDL (deep in deep learning) önerilmiştir. Önerilen DDL yaklaşımının performansı, Kayseri Büyükşehir Belediyesi (KBB) Yeşilhisar ilçesinin 2019 yılına ait GPS konum bilgisiyle fotoğrafı çekilen 35 adet mahallenin bina sokak görüntülerinden oluşan 113 GB (gigabayt) boyuta sahip 17.618 adet görüntü içeren gerçek veriler kullanılarak, DL yaklaşımının performansıyla karşılaştırılmıştır. Deneysel sonuçlar, önerilen DDL yaklaşımının DL yaklaşımına göre daha doğru sonuçlar ürettiğini ve daha az depolama alanı kullandığını göstermektedir.

Anahtar Kelimeler

• Evrişimsel sinir ağı
• Derin öğrenme
• Çoklu kapı numarası tespiti ve kümeleme

Kaynakça

1. Batuk, F., Öztürk, D., Emem, O., Türkiye Ulusal Konumsal Veri Altyapısı İçin Temel Veriler. Jeodezi ve Jeoinformasyon Dergisi, (96), 3-12, 2007.
2. Goodfellow, I. J., Bulatov, Y., Ibarz, J., Arnoud, S., Shet, V., Multi-digit number recognition from street view imagery using deep convolutional neural networks, arXiv preprint arXiv:1312.6082, 2013.
3. Türk, T., Adres Kayıt Sistemi ile Kent Bilgi Sistemlerinin Bütünleştirilmesi, Jeodezi ve Jeoinformasyon Dergisi, (99), 13-22, 2008.
4. Anguelov, D., Dulong, C., Filip, D., Frueh, C., Lafon, S., Lyon, R., ... Weaver, J., Google street view: Capturing the world at street level, Computer, 43(6), 32-38, 2010.
5. Bayram, F., Derin öğrenme tabanlı otomatik plaka tanıma, Politeknik Dergisi, 23(4), 955-960, 2020.
6. Zuo, L. Q., Sun, H. M., Mao, Q. C., Qi, R., Jia, R. S., Natural scene text recognition based on encoder-decoder framework, IEEE Access, 7, 62616-62623, 2019.
7. Aktaş, A., Doğan, B., Demi̇r, Ö., Derin öğrenme yöntemleri ile dokunsal parke yüzeyi tespiti, Journal of the Faculty of Engineering and Architecture of Gazi University, 35 (3), 1685-1700, 2020.
8. Perez, L., Wang, J., The effectiveness of data augmentation in image classification using deep learning, arXiv preprint arXiv:1712.04621, 2017.

9. Guo, T., Dong, J., Li, H., Gao, Y., Simple convolutional neural network on image classification, 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA), IEEE, 721-724, 2017.
10. Lu, L., Zheng, Y., Carneiro, G., Yang, L., Deep learning and convolutional neural networks for medical image computing, Advances in Computer Vision and Pattern Recognition, 10, 978-3, 2017.
11. Albawi, S., Mohammed, T. A., Al-Zawi, S., Understanding of a convolutional neural network, 2017 International Conference on Engineering and Technology (ICET), IEEE, 1-6, 2017.
12. Kalchbrenner, N., Grefenstette, E., Blunsom, P., A convolutional neural network for modelling sentences, arXiv preprint arXiv:1404.2188, 2014.
13. Li, Y. D., Hao, Z. B., Lei, H., Survey of convolutional neural network, Journal of Computer Applications, 36(9), 2508-2515, 2016.
14. Vinayakumar, R., Soman, K. P., Poornachandran, P., Applying convolutional neural network for network intrusion detection, 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, 1222-1228, 2017.
15. Chauhan, R., Ghanshala, K. K., Joshi, R. C., Convolutional neural network (CNN) for image detection and recognition, 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), IEEE, 278-282, 2018.
16. Kido, S., Hirano, Y., & Hashimoto, N., Detection and classification of lung abnormalities by use of convolutional neural network (CNN) and regions with CNN features (R-CNN), In 2018 International workshop on advanced image technology (IWAIT), IEEE, 1-4, 2018.
17. Özcan, T., Baştürk, A. ERUSLR: Yeni bir Türkçe işaret dili veri seti ve hiperparametre optimizasyonu destekli evrişimli sinir ağı ile tanınması, Journal of the Faculty of Engineering and Architecture of Gazi University, 36(1), 527-542, 2020.
18. Sonmez, E. B., yıldız, T., yılmaz, B. D., Demir, A. E., Türkçe dilinde görüntü altyazısı: veritabanı ve model, Journal of the Faculty of Engineering and Architecture of Gazi University, 35(4), 2089-2100, 2020.
19. Kagaya, H., Aizawa, K., Ogawa, M., Food detection and recognition using convolutional neural network, Proceedings of the 22nd ACM international conference on Multimedia, 1085-1088, 2014.
20. Hansen, M. F., Smith, M. L., Smith, L. N., Salter, M. G., Baxter, E. M., Farish, M., Grieve, B., Towards on-farm pig face recognition using convolutional neural networks, Computers in Industry, 98, 145-152, 2018.
21. Pramerdorfer, C., & Kampel, M., Facial expression recognition using convolutional neural networks: state of the art, arXiv preprint arXiv:1612.02903, 2016.
22. Gerke, S., Muller, K., & Schafer, R., Soccer jersey number recognition using convolutional neural networks, Proceedings of the IEEE International Conference on Computer Vision Workshops, 17-24, 2015.
23. Lorencin, I., Anđelić, N., Mrzljak, V., Car, Z., Marine objects recognition using convolutional neural networks, NAŠE MORE: znanstveni časopis za more i pomorstvo, 66(3), 112-119, 2019.
24. Li, Q., Cai, W., Wang, X., Zhou, Y., Feng, D. D., Chen, M., Medical image classification with convolutional neural network, 2014 13th international conference on control automation robotics & vision (ICARCV), IEEE, 844-848, 2014.
25. Alwzwazy, H. A., Albehadili, H. M., Alwan, Y. S., Islam, N. E., Handwritten digit recognition using convolutional neural networks, International Journal of Innovative Research in Computer and Communication Engineering, 4(2), 1101-1106, 2016.
26. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86(11), 2278-2324, 1998.
27. El-Sawy, A., Hazem, E. B., Loey, M., CNN for handwritten arabic digits recognition based on LeNet-5, In International conference on advanced intelligent systems and informatics, Springer, Cham, 566-575, 2016.
28. Silaparasetty, V., Neural Network Collection, Deep Learning Projects Using TensorFlow 2, Berkeley, CA, 249-347, Apress, 2020.
29. Krizhevsky, A., Sutskever, I., Hinton, G. E., Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25, 1097-1105, 2012.
30. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., ... Murphy, K., Speed/accuracy trade-offs for modern convolutional object detectors, Proceedings of the IEEE conference on computer vision and pattern recognition, 7310-7311, 2017.
31. Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.
32. He, K., Zhang, X., Ren, S., & Sun, J., Deep residual learning for image recognition, Proceedings of the IEEE conference on computer vision and pattern recognition, IEEE, 770-778, 2016.
33. Girshick, R., Donahue, J., Darrell, T., Malik, J., Rich feature hierarchies for accurate object detection and semantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition, IEEE, 580-587, 2014.
34. Girshick, R., Fast r-cnn, Proceedings of the IEEE international conference on computer vision, IEEE, 1440-1448, 2015.
35. Redmon, J., Divvala, S., Girshick, R., Farhadi, A., You only look once: Unified, real-time object detection, Proceedings of the IEEE conference on computer vision and pattern recognition, IEEE, 779-788, 2016.
36. Huang, R., Pedoeem, J., Chen, C., YOLO-LITE: a real-time object detection algorithm optimized for non-GPU computers, 2018 IEEE International Conference on Big Data (Big Data), IEEE, 2503-2510, 2018.
37. Du, J., Understanding of object detection based on CNN family and YOLO, Journal of Physics: Conference Series, IOP Publishing, 1004 (1), 012029, 2018.
38. Laroca, R., Severo, E., Zanlorensi, L. A., Oliveira, L. S., Gonçalves, G. R., Schwartz, W. R., Menotti, D., A robust real-time automatic license plate recognition based on the YOLO detector, 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, 1-10, 2018.
39. Tao, J., Wang, H., Zhang, X., Li, X., Yang, H., An object detection system based on YOLO in traffic scene, 2017 6th International Conference on Computer Science and Network Technology (ICCSNT), IEEE, 315-319, 2017.
40. Adarsh, P., Rathi, P., Kumar, M., YOLO v3-Tiny: Object Detection and Recognition using one stage improved model, 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), IEEE, 687-694, 2020.
41. Wang, D., Li, C., Wen, S., Han, Q. L., Nepal, S., Zhang, X., Xiang, Y., Daedalus: Breaking nonmaximum suppression in object detection via adversarial examples, IEEE Transactions on Cybernetics.
42. Redmon, J., Farhadi, A., YOLO9000: better, faster, stronger, Proceedings of the IEEE conference on computer vision and pattern recognition, IEEE, 7263-7271, 2017.
43. Redmon, J., Farhadi, A., Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767, 2018.
44. Bochkovskiy, A., Wang, C. Y., Liao, H. Y. M., Yolov4: Optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.109, 2020.
45. Bisong, E., Google colaboratory, Building Machine Learning and Deep Learning Models on Google Cloud Platform, Apress, Berkeley, CA, 59-64, 2019.
46. Wang, S., Niu, L., Li, N., Research on image recognition of insulators based on YOLO algorithm, 2018 international conference on power system technology (POWERCON), IEEE, 3871-3874, 2018.
47. Clark, A., Pillow (PIL fork) documentation, 2015.