Classification of Scatter Plot Images Using Deep Learning

Year 2022, , 631 - 642, 16.05.2022

Derya Bırant , Aydanur Akça , Buse Bozkurt , Mehtap Bağlan

https://doi.org/10.21205/deufmd.2022247126

Abstract

Scatter plot is one of the well-known charts and is frequently embedded in different types of documents such as articles, books, and dissertations. However, the information given in the scatter plots can’t be directly noticed by visually impaired individuals, because they are usually in an image format, and so they are not naturally readable by machines. To solve this problem, this paper proposes a system that can extract visual properties from scatter plot images using deep learning and image processing techniques. It is the first study that automatically classifies scatter plots in terms of two aspects: degree of correlation (strong or weak) and types of correlation (positive, negative, or neutral). In the experimental studies, alternative convolutional neural network (CNN) architectures were compared on both synthetic and real-world datasets in terms of accuracy, including Residual Networks (ResNet), Alex Networks (AlexNet), and Visual Geometry Group (VGG) Networks. The experimental results showed that the proposed system successfully (93.90%) classified scatter plot images to help visually impaired users understand the information given in the graph.

Keywords

Scatter plots, Image classification, Deep learning, Machine learning

Derin Öğrenme Kullanarak Dağılım Grafiği Görüntülerinin Sınıflandırılması

Abstract

Dağılım grafiği, iyi bilinen grafiklerden biridir ve makaleler, kitaplar, raporlar gibi birçok farklı türdeki dokümanlarda sıklıkla kullanılmaktadır. Ancak, dağılım grafikleri genellikle görüntü biçiminde olduğu için grafiklerde verilen bilgiler görme engelli kişiler tarafından fark edilemez, yani esasen makine tarafından okunabilir değillerdir. Bu sorunu çözmek için, bu makale, derin öğrenme ve görüntü işleme tekniklerini kullanarak, dağılım grafiği görüntülerinden görsel özellikleri çıkartabilen bir sistem önermektedir. Dağılım grafiklerini iki açıdan otomatik olarak sınıflandıran ilk çalışmadır: korelasyon derecesi (güçlü veya zayıf) ve korelasyon türleri (pozitif, negatif veya nötr). Deneysel çalışmalarda, Artık Ağlar (ResNet), Alex Ağları (AlexNet) ve Görsel Geometri Grubu (VGG) Ağları gibi alternatif evrişimsel sinir ağı (CNN) mimarileri hem sentetik hem de gerçek dünya veri setlerinde doğruluk açısından karşılaştırılmıştır. Deneysel sonuçlar, önerilen sistemin başarılı bir şekilde (%93,90) dağılım grafiği görüntülerini sınıflandırarak görme engelli kullanıcıların grafikte verilen bilgileri anlamalarına yardımcı olduğunu göstermiştir.

Keywords

Dağılım grafikleri, Görüntü sınıflandırma, Derin öğrenme, Makine öğrenmesi

References

• Shao, L., Mahajan, A., Schreck, T., Lehmann, D.J. 2017. Interactive Regression Lens for Exploring Scatter Plots, Computer Graphics Forum, Volume. 36, p. 157-166. DOI: 10.1111/cgf.13176
• Wang, W.B., Huang, M.L., Nguyen, Q.V., Huang, W., Zhang, K., Huang, T.H. 2016. Enabling Decision Trend Analysis with Interactive Scatter Plot Matrices Visualization, Journal of Visual Languages & Computing, Volume. 33, p. 13-23. DOI: 10.1016/j.jvlc.2015.11.002
• Sainani, K.L. 2016. The Value of Scatter Plots, Physical Medicine and Rehabilitation (PM&R), Volume. 8, p. 1213-1217. DOI: 10.1016/j.pmrj.2016.10.018
• Mohseni, F., Mokhtarzade, M. 2021. The Synergistic Use of Microwave Coarse-scale Measurements and Two Adopted High-resolution Indices Driven from Long-term T-V Scatter Plot for Fine-scale Soil Moisture Estimation, GIScience & Remote Sensing, Volume. 58, p. 455-482. DOI: 10.1080/15481603.2021.1906056
• Zhang, Z., Cui, X., Jeske, D.R., Borneman, J. 2013. Biclustering Scatter Plots Using Data Depth Measures, Statistical Analysis and Data Mining, Volume. 6, p. 102-115. DOI: 10.1002/sam.11166
• Fu, J., Zhu, B., Cui, W., Ge, S., Wang, Y., Zhang, H., Huang, H., Tang, Y., Zhang, D., Ma, X. 2020. Chartem: Reviving Chart Images with Data Embedding, IEEE Transactions on Visualization and Computer Graphics, Volume. 27, p. 337-346. DOI: 10.1109/TVCG.2020.3030351
• Al-Zaidy, R.A., Giles, C.L. 2015. Automatic Extraction of Data from Bar Charts. 8th International Conference on Knowledge Capture, 07-10 October, Palisades, USA, 1-4. DOI: 10.1145/2815833.2816956
• Bajic, F., Job, J., Nenadic, K. 2019. Chart Classification Using Simplified VGG Model, International Conference on Systems, Signals and Image Processing, 5–7 June, Osijek, Croatia, 229-233. DOI: 10.1109/IWSSIP.2019.8787299
• Araujo, T., Chagas, P., Alves, J., Santos, C., Santos, B., Meiguins, B. 2020. A Real-World Approach on the Problem of Chart Recognition Using Classification, Detection and Perspective Correction, Sensors, Volume. 20, p. 1-21. DOI: 10.3390/s20164370
• Deepa, R., Tamilselvan, E. 2020. Processing Of Evaluation Chart Uses Optical Character Recognition, International Journal of Scientific & Technology Research, Volume. 9, p. 4770-4773.
• Vassilieva, N., Fomina, Y. 2013. Text Detection in Chart Images, Pattern Recognition and Image Analysis, Volume. 23, p. 139-144. DOI: 10.1134/S1054661813010112
• Chagas, P., Freitas, A., Daisuke, R., Miranda, B., De Araújo, T.D.O., Santos, C., Meiguins, B., De Morais, J.M. 2017. Architecture Proposal for Data Extraction of Chart Images Using Convolutional Neural Network, 21st International Conference on Information Visualisation, 11–14 July, London, UK, 318-323. DOI: 10.1109/iV.2017.37
• Zan, T., Liu, Z., Wang, H., Wang, M., Gao, X. 2020. Control Chart Pattern Recognition Using the Convolutional Neural Network, Journal of Intelligent Manufacturing, Volume. 31, p. 703-716. DOI: 10.1007/s10845-019-01473-0
• Mishra, P., Kumar, S., Chaube, M.K. 2020. Dissimilarity Based Regularized Deep Learning Model for Information Charts, 9th International Conference on Informatics, Electronics & Vision, 26-29 August, Kitakyushu, Japan, 1-6. DOI: 10.1109/ICIEVicIVPR48672.2020.9306660
• Zhou, F., Zhao, Y., Chen, W., Tan, Y., Xu, Y., Chen, Y., Liu, C., Zhao, Y. 2021. Reverse-engineering Bar Charts Using Neural Networks. Journal of Visualization, Volume. 24, p. 419-435. DOI: 10.1007/s12650-020-00702-6
• Sohn, C., Choi, H., Kim, K., Park, J., Noh, J. 2021. Line Chart Understanding with Convolutional Neural Network, Electronics, Volume. 10, p. 1-17. DOI: 10.3390/electronics10060749
• De, P. 2018. Automatic Data Extraction from 2D and 3D Pie Chart Images, IEEE 8th International Advance Computing Conference, 14–15 December, India, 20-25. DOI: 10.1109/IADCC.2018.8692104
• Fuqua, D., Razzaghi, T. 2020. A Cost-sensitive Convolution Neural Network Learning for Control Chart Pattern Recognition, Expert Systems with Applications, Volume. 150, p. 1-17. DOI: 10.1016/j.eswa.2020.113275
• Zan, T., Su, Z., Liu, Z., Chen, D., Wang, M., Gao, X. 2020. Pattern Recognition of Different Window Size Control Charts Based on Convolutional Neural Network and Information Fusion, Symmetry, Volume. 12, p. 1-13. DOI: 10.3390/sym13010110
• Birogul, S., Temür, G., Kose, U. 2020. YOLO Object Recognition Algorithm and “Buy-Sell Decision” Model Over 2D Candlestick Charts, IEEE Access, Volume. 8, p. 91894-91915. DOI: 10.1109/ACCESS.2020.2994282
• Siper, M., Makinen, K., Kanan, R. 2021. TABot – A Distributed Deep Learning Framework for Classifying Price Chart Images, Advanced Computing, Volume. 1367. p. 1-15. DOI: 10.1007/978-981-16-0401-0_37
• Ünlü, R. 2021. A Robust Data Simulation Technique to Improve Early Detection Performance of a Classifier in Control Chart Pattern Recognition Systems, Information Sciences, Volume. 548, p. 18-36. DOI: 10.1016/j.ins.2020.09.059
• Kaur, P., Kiesel, D. 2020. Combining Image and Caption Analysis for Classifying Charts in Biodiversity Texts, 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 27-29 February, Valletta, Malta, 157-168. DOI: 10.5220/0008946701570168
• Zaman, M., Hassan, A. 2021. Fuzzy Heuristics and Decision Tree for Classification of Statistical Feature-Based Control Chart Patterns, Symmetry, Volume. 13, p. 1-12. DOI: 10.3390/sym13010110
• Mishra, P. & Kumar, S., Chaube, M.K. 2020. Interpretation and Segmentation of Chart Images Using h-Means Image Clustering Algorithm, International Conference on Data Management, Analytics and Innovation, 17–19 January, New Delhi, India, 379-391. DOI: 10.1007/978-981-15-5616-6_27
• Sharma, M., Gupta, S., Chowdhury, A., Vig, L. 2019. ChartNet: Visual Reasoning Over Statistical Charts Using MAC-Networks, International Joint Conference on Neural Networks, 14-19 July, Budapest, Hungary, 1-7. DOI: 10.1109/IJCNN.2019.8852427
• Mishchenko, A., Vassilieva, N. 2011. Chart Image Understanding and Numerical Data Extraction, 6th International Conference on Digital Information Management, 26–28 September, Australia, 115-210. DOI: 10.1109/ICDIM.2011.6093320
• Mishchenko, A., Vassilieva, N. 2011. Model-based Chart Image Classification, International Symposium on Visual Computing, Advances in Visual Computing, Lecture Notes in Computer Science, Volume. 6939, p. 476-485. DOI: 10.1007/978-3-642-24031 -7_48
• Maboudou-Tchao, E.M. 2020. Change Detection Using Least Squares One-class Classification Control Chart, Quality Technology & Quantitative Management, Volume. 17, p. 609-626. DOI: 10.1080/16843703.2019.1711302
• Mishra, P., Kumar, S., Chaube, M.K. 2020. ChartFuse: A Novel Fusion Method for Chart Classification Using Heterogeneous Microstructures, Multimedia Tools and Applications, Volume. 80, p. 10417-10439. DOI: 10.1007/s11042-020-10186-z
• Direncioğlu Diren, D., Boran, S., Cil, I. 2020. Integration of Machine Learning Techniques and Control Charts in Multivariate Processes, Scientia Iranica, Volume. 27, p. 3233-3241, DOI: 10.24200/sci.2019.50377.1667
• Kalteh, A.A., Babouei, S. 2020. Control Chart Patterns Recognition Using ANFIS with New Training Algorithm and Intelligent Utilization of Shape and Statistical Features, ISA Transactions, Volume. 102, p. 12-22. DOI: 10.1016/j.isatra.2019.12.001
• Kadakadiyavar, S., Ramrao, N., Singh, M.K. 2019. Efficient Mixture Control Chart Pattern Recognition Using Adaptive RBF Neural Network, International Journal of Information Technology, Volume. 12, p. 1271-1280. DOI: 10.1007/s41870-019-00381-z
• Shao, Y.E., Hu, Y.T. 2020. Using Machine Learning Classifiers to Recognize the Mixture Control Chart Patterns for a Multiple-input Multiple-output Process, Mathematics, Volume. 8, p. 1-14. DOI: 10.3390/math8010102
• Dai, W., Wang, M., Niu, Z., Zhang, J. 2018. Chart Decoder: Generating Textual and Numeric Information from Chart Images Automatically, Journal of Visual Languages & Computing, Volume. 48, p. 101-109. DOI: 10.1016/j.jvlc.2018.08.005
• Tang, B., Liu, X., Lei, J., Song, M., Tao, D., Sun, S., Dong, F. 2016. DeepChart: Combining Deep Convolutional Networks and Deep Belief Networks in Chart Classification, Signal Process, Volume. 124, p. 156-161. DOI: 10.1016/j.sigpro.2015.09.027
• Singh, M., Goyal, P. 2021. ChartSight: An Automated Scheme for Assisting Visually Impaired in Understanding Scientific Charts, 16th International Joint Conference on Computer Vision, 8-10 February, Setubal, Portugal, 309-318. DOI: 10.5220/0010201203090318
• Savva, M., Kong, N., Chhajta, A., Li, F.F., Agrawala, M., Heer, J. 2011. ReVision: Automated Classification, Analysis and Redesign of Chart Images, 24th Annual ACM Symposium on User Interface Software and Technology, 16-19 October, California, USA, 393-402. DOI: 10.1145/20471 96.20472 47
• Jung, D., Kim, W., Song, H., Hwang, J., Lee, B., Kim, B., Seo, J. 2017. ChartSense: Interactive Data Extraction from Chart Images, Conference on Human Factors in Computing Systems, 6-11 May, Denver Colorado, USA, 6706-6717. DOI: 10.1145/30254 53.30259 57
• Krizhevsky, A., Sutskever, I., Hinton, G.E. 2012. ImageNet Classification with Deep Convolutional Neural Networks, International Conference on Neural Information Processing Systems, 3-6 December, Nevada, USA, 1097–1105.
• Simonyan, K., Zisserman, A. 2015. Very Deep Convolutional Networks for Large-scale Image Recognition, 3rd International Conference on Learning Representations, 7-9 May, San Diego, CA, USA, 1–14.
• He, K., Zhang, X., Ren, S., Sun, J. 2016. Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 26 June-1 July, Las Vegas, USA, 770-778. DOI: 10.1109/CVPR.2016.90
• Jia, S., Wang, P., Jia, P., Hu, S. 2017. Research on Data Augmentation for Image Classification Based on Convolution Neural Networks. Chinese Automation Congress, 20-22 October, Jinan, China, 4165-4170. DOI: 10.1109/CAC.2017.8243510
• Wicaksono P. 2016. Improving the Accuracy of Multispectral-based Benthic Habitats Mapping Using Image Rotations: The Application of Principle Component Analysis and Independent Component Analysis, European Journal of Remote Sensing, Volume. 49, p. 433-463. DOI: 10.5721/EuJRS20164924
• Liang, G., Hong, H., Xie, W., Zheng L. 2018. Combining Convolutional Neural Network With Recursive Neural Network for Blood Cell Image Classification, IEEE Access, Volume. 6, p. 36188-36197. DOI: 10.1109/ACCESS.2018.2846685
• Somasundaram, D. 2019. Machine Learning Approach for Homolog Chromosome Classification, International Journal of Imaging Systems and Technology, Volume. 29, p. 161-167. DOI: 10.1002/ima.22287