Classification of Scatter Plot Images Using Deep Learning

Abstract

Scatter plot is one of the well-known charts and is frequently embedded in different types of documents such as articles, books, and dissertations. However, the information given in the scatter plots can’t be directly noticed by visually impaired individuals, because they are usually in an image format, and so they are not naturally readable by machines. To solve this problem, this paper proposes a system that can extract visual properties from scatter plot images using deep learning and image processing techniques. It is the first study that automatically classifies scatter plots in terms of two aspects: degree of correlation (strong or weak) and types of correlation (positive, negative, or neutral). In the experimental studies, alternative convolutional neural network (CNN) architectures were compared on both synthetic and real-world datasets in terms of accuracy, including Residual Networks (ResNet), Alex Networks (AlexNet), and Visual Geometry Group (VGG) Networks. The experimental results showed that the proposed system successfully (93.90%) classified scatter plot images to help visually impaired users understand the information given in the graph.

Keywords

• Scatter plots
• Image classification
• Deep learning
• Machine learning

Derin Öğrenme Kullanarak Dağılım Grafiği Görüntülerinin Sınıflandırılması

Abstract

Dağılım grafiği, iyi bilinen grafiklerden biridir ve makaleler, kitaplar, raporlar gibi birçok farklı türdeki dokümanlarda sıklıkla kullanılmaktadır. Ancak, dağılım grafikleri genellikle görüntü biçiminde olduğu için grafiklerde verilen bilgiler görme engelli kişiler tarafından fark edilemez, yani esasen makine tarafından okunabilir değillerdir. Bu sorunu çözmek için, bu makale, derin öğrenme ve görüntü işleme tekniklerini kullanarak, dağılım grafiği görüntülerinden görsel özellikleri çıkartabilen bir sistem önermektedir. Dağılım grafiklerini iki açıdan otomatik olarak sınıflandıran ilk çalışmadır: korelasyon derecesi (güçlü veya zayıf) ve korelasyon türleri (pozitif, negatif veya nötr). Deneysel çalışmalarda, Artık Ağlar (ResNet), Alex Ağları (AlexNet) ve Görsel Geometri Grubu (VGG) Ağları gibi alternatif evrişimsel sinir ağı (CNN) mimarileri hem sentetik hem de gerçek dünya veri setlerinde doğruluk açısından karşılaştırılmıştır. Deneysel sonuçlar, önerilen sistemin başarılı bir şekilde (%93,90) dağılım grafiği görüntülerini sınıflandırarak görme engelli kullanıcıların grafikte verilen bilgileri anlamalarına yardımcı olduğunu göstermiştir.

Keywords

• Dağılım grafikleri
• Görüntü sınıflandırma
• Derin öğrenme
• Makine öğrenmesi

References

1. Shao, L., Mahajan, A., Schreck, T., Lehmann, D.J. 2017. Interactive Regression Lens for Exploring Scatter Plots, Computer Graphics Forum, Volume. 36, p. 157-166. DOI: 10.1111/cgf.13176
2. Wang, W.B., Huang, M.L., Nguyen, Q.V., Huang, W., Zhang, K., Huang, T.H. 2016. Enabling Decision Trend Analysis with Interactive Scatter Plot Matrices Visualization, Journal of Visual Languages & Computing, Volume. 33, p. 13-23. DOI: 10.1016/j.jvlc.2015.11.002
3. Sainani, K.L. 2016. The Value of Scatter Plots, Physical Medicine and Rehabilitation (PM&R), Volume. 8, p. 1213-1217. DOI: 10.1016/j.pmrj.2016.10.018
4. Mohseni, F., Mokhtarzade, M. 2021. The Synergistic Use of Microwave Coarse-scale Measurements and Two Adopted High-resolution Indices Driven from Long-term T-V Scatter Plot for Fine-scale Soil Moisture Estimation, GIScience & Remote Sensing, Volume. 58, p. 455-482. DOI: 10.1080/15481603.2021.1906056
5. Zhang, Z., Cui, X., Jeske, D.R., Borneman, J. 2013. Biclustering Scatter Plots Using Data Depth Measures, Statistical Analysis and Data Mining, Volume. 6, p. 102-115. DOI: 10.1002/sam.11166
6. Fu, J., Zhu, B., Cui, W., Ge, S., Wang, Y., Zhang, H., Huang, H., Tang, Y., Zhang, D., Ma, X. 2020. Chartem: Reviving Chart Images with Data Embedding, IEEE Transactions on Visualization and Computer Graphics, Volume. 27, p. 337-346. DOI: 10.1109/TVCG.2020.3030351
7. Al-Zaidy, R.A., Giles, C.L. 2015. Automatic Extraction of Data from Bar Charts. 8th International Conference on Knowledge Capture, 07-10 October, Palisades, USA, 1-4. DOI: 10.1145/2815833.2816956
8. Bajic, F., Job, J., Nenadic, K. 2019. Chart Classification Using Simplified VGG Model, International Conference on Systems, Signals and Image Processing, 5–7 June, Osijek, Croatia, 229-233. DOI: 10.1109/IWSSIP.2019.8787299

9. Araujo, T., Chagas, P., Alves, J., Santos, C., Santos, B., Meiguins, B. 2020. A Real-World Approach on the Problem of Chart Recognition Using Classification, Detection and Perspective Correction, Sensors, Volume. 20, p. 1-21. DOI: 10.3390/s20164370
10. Deepa, R., Tamilselvan, E. 2020. Processing Of Evaluation Chart Uses Optical Character Recognition, International Journal of Scientific & Technology Research, Volume. 9, p. 4770-4773.
11. Vassilieva, N., Fomina, Y. 2013. Text Detection in Chart Images, Pattern Recognition and Image Analysis, Volume. 23, p. 139-144. DOI: 10.1134/S1054661813010112
12. Chagas, P., Freitas, A., Daisuke, R., Miranda, B., De Araújo, T.D.O., Santos, C., Meiguins, B., De Morais, J.M. 2017. Architecture Proposal for Data Extraction of Chart Images Using Convolutional Neural Network, 21st International Conference on Information Visualisation, 11–14 July, London, UK, 318-323. DOI: 10.1109/iV.2017.37
13. Zan, T., Liu, Z., Wang, H., Wang, M., Gao, X. 2020. Control Chart Pattern Recognition Using the Convolutional Neural Network, Journal of Intelligent Manufacturing, Volume. 31, p. 703-716. DOI: 10.1007/s10845-019-01473-0
14. Mishra, P., Kumar, S., Chaube, M.K. 2020. Dissimilarity Based Regularized Deep Learning Model for Information Charts, 9th International Conference on Informatics, Electronics & Vision, 26-29 August, Kitakyushu, Japan, 1-6. DOI: 10.1109/ICIEVicIVPR48672.2020.9306660
15. Zhou, F., Zhao, Y., Chen, W., Tan, Y., Xu, Y., Chen, Y., Liu, C., Zhao, Y. 2021. Reverse-engineering Bar Charts Using Neural Networks. Journal of Visualization, Volume. 24, p. 419-435. DOI: 10.1007/s12650-020-00702-6
16. Sohn, C., Choi, H., Kim, K., Park, J., Noh, J. 2021. Line Chart Understanding with Convolutional Neural Network, Electronics, Volume. 10, p. 1-17. DOI: 10.3390/electronics10060749
17. De, P. 2018. Automatic Data Extraction from 2D and 3D Pie Chart Images, IEEE 8th International Advance Computing Conference, 14–15 December, India, 20-25. DOI: 10.1109/IADCC.2018.8692104
18. Fuqua, D., Razzaghi, T. 2020. A Cost-sensitive Convolution Neural Network Learning for Control Chart Pattern Recognition, Expert Systems with Applications, Volume. 150, p. 1-17. DOI: 10.1016/j.eswa.2020.113275
19. Zan, T., Su, Z., Liu, Z., Chen, D., Wang, M., Gao, X. 2020. Pattern Recognition of Different Window Size Control Charts Based on Convolutional Neural Network and Information Fusion, Symmetry, Volume. 12, p. 1-13. DOI: 10.3390/sym13010110
20. Birogul, S., Temür, G., Kose, U. 2020. YOLO Object Recognition Algorithm and “Buy-Sell Decision” Model Over 2D Candlestick Charts, IEEE Access, Volume. 8, p. 91894-91915. DOI: 10.1109/ACCESS.2020.2994282
21. Siper, M., Makinen, K., Kanan, R. 2021. TABot – A Distributed Deep Learning Framework for Classifying Price Chart Images, Advanced Computing, Volume. 1367. p. 1-15. DOI: 10.1007/978-981-16-0401-0_37
22. Ünlü, R. 2021. A Robust Data Simulation Technique to Improve Early Detection Performance of a Classifier in Control Chart Pattern Recognition Systems, Information Sciences, Volume. 548, p. 18-36. DOI: 10.1016/j.ins.2020.09.059
23. Kaur, P., Kiesel, D. 2020. Combining Image and Caption Analysis for Classifying Charts in Biodiversity Texts, 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 27-29 February, Valletta, Malta, 157-168. DOI: 10.5220/0008946701570168
24. Zaman, M., Hassan, A. 2021. Fuzzy Heuristics and Decision Tree for Classification of Statistical Feature-Based Control Chart Patterns, Symmetry, Volume. 13, p. 1-12. DOI: 10.3390/sym13010110
25. Mishra, P. & Kumar, S., Chaube, M.K. 2020. Interpretation and Segmentation of Chart Images Using h-Means Image Clustering Algorithm, International Conference on Data Management, Analytics and Innovation, 17–19 January, New Delhi, India, 379-391. DOI: 10.1007/978-981-15-5616-6_27
26. Sharma, M., Gupta, S., Chowdhury, A., Vig, L. 2019. ChartNet: Visual Reasoning Over Statistical Charts Using MAC-Networks, International Joint Conference on Neural Networks, 14-19 July, Budapest, Hungary, 1-7. DOI: 10.1109/IJCNN.2019.8852427
27. Mishchenko, A., Vassilieva, N. 2011. Chart Image Understanding and Numerical Data Extraction, 6th International Conference on Digital Information Management, 26–28 September, Australia, 115-210. DOI: 10.1109/ICDIM.2011.6093320
28. Mishchenko, A., Vassilieva, N. 2011. Model-based Chart Image Classification, International Symposium on Visual Computing, Advances in Visual Computing, Lecture Notes in Computer Science, Volume. 6939, p. 476-485. DOI: 10.1007/978-3-642-24031 -7_48
29. Maboudou-Tchao, E.M. 2020. Change Detection Using Least Squares One-class Classification Control Chart, Quality Technology & Quantitative Management, Volume. 17, p. 609-626. DOI: 10.1080/16843703.2019.1711302
30. Mishra, P., Kumar, S., Chaube, M.K. 2020. ChartFuse: A Novel Fusion Method for Chart Classification Using Heterogeneous Microstructures, Multimedia Tools and Applications, Volume. 80, p. 10417-10439. DOI: 10.1007/s11042-020-10186-z
31. Direncioğlu Diren, D., Boran, S., Cil, I. 2020. Integration of Machine Learning Techniques and Control Charts in Multivariate Processes, Scientia Iranica, Volume. 27, p. 3233-3241, DOI: 10.24200/sci.2019.50377.1667
32. Kalteh, A.A., Babouei, S. 2020. Control Chart Patterns Recognition Using ANFIS with New Training Algorithm and Intelligent Utilization of Shape and Statistical Features, ISA Transactions, Volume. 102, p. 12-22. DOI: 10.1016/j.isatra.2019.12.001
33. Kadakadiyavar, S., Ramrao, N., Singh, M.K. 2019. Efficient Mixture Control Chart Pattern Recognition Using Adaptive RBF Neural Network, International Journal of Information Technology, Volume. 12, p. 1271-1280. DOI: 10.1007/s41870-019-00381-z
34. Shao, Y.E., Hu, Y.T. 2020. Using Machine Learning Classifiers to Recognize the Mixture Control Chart Patterns for a Multiple-input Multiple-output Process, Mathematics, Volume. 8, p. 1-14. DOI: 10.3390/math8010102
35. Dai, W., Wang, M., Niu, Z., Zhang, J. 2018. Chart Decoder: Generating Textual and Numeric Information from Chart Images Automatically, Journal of Visual Languages & Computing, Volume. 48, p. 101-109. DOI: 10.1016/j.jvlc.2018.08.005
36. Tang, B., Liu, X., Lei, J., Song, M., Tao, D., Sun, S., Dong, F. 2016. DeepChart: Combining Deep Convolutional Networks and Deep Belief Networks in Chart Classification, Signal Process, Volume. 124, p. 156-161. DOI: 10.1016/j.sigpro.2015.09.027
37. Singh, M., Goyal, P. 2021. ChartSight: An Automated Scheme for Assisting Visually Impaired in Understanding Scientific Charts, 16th International Joint Conference on Computer Vision, 8-10 February, Setubal, Portugal, 309-318. DOI: 10.5220/0010201203090318
38. Savva, M., Kong, N., Chhajta, A., Li, F.F., Agrawala, M., Heer, J. 2011. ReVision: Automated Classification, Analysis and Redesign of Chart Images, 24th Annual ACM Symposium on User Interface Software and Technology, 16-19 October, California, USA, 393-402. DOI: 10.1145/20471 96.20472 47
39. Jung, D., Kim, W., Song, H., Hwang, J., Lee, B., Kim, B., Seo, J. 2017. ChartSense: Interactive Data Extraction from Chart Images, Conference on Human Factors in Computing Systems, 6-11 May, Denver Colorado, USA, 6706-6717. DOI: 10.1145/30254 53.30259 57
40. Krizhevsky, A., Sutskever, I., Hinton, G.E. 2012. ImageNet Classification with Deep Convolutional Neural Networks, International Conference on Neural Information Processing Systems, 3-6 December, Nevada, USA, 1097–1105.
41. Simonyan, K., Zisserman, A. 2015. Very Deep Convolutional Networks for Large-scale Image Recognition, 3rd International Conference on Learning Representations, 7-9 May, San Diego, CA, USA, 1–14.
42. He, K., Zhang, X., Ren, S., Sun, J. 2016. Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 26 June-1 July, Las Vegas, USA, 770-778. DOI: 10.1109/CVPR.2016.90
43. Jia, S., Wang, P., Jia, P., Hu, S. 2017. Research on Data Augmentation for Image Classification Based on Convolution Neural Networks. Chinese Automation Congress, 20-22 October, Jinan, China, 4165-4170. DOI: 10.1109/CAC.2017.8243510
44. Wicaksono P. 2016. Improving the Accuracy of Multispectral-based Benthic Habitats Mapping Using Image Rotations: The Application of Principle Component Analysis and Independent Component Analysis, European Journal of Remote Sensing, Volume. 49, p. 433-463. DOI: 10.5721/EuJRS20164924
45. Liang, G., Hong, H., Xie, W., Zheng L. 2018. Combining Convolutional Neural Network With Recursive Neural Network for Blood Cell Image Classification, IEEE Access, Volume. 6, p. 36188-36197. DOI: 10.1109/ACCESS.2018.2846685
46. Somasundaram, D. 2019. Machine Learning Approach for Homolog Chromosome Classification, International Journal of Imaging Systems and Technology, Volume. 29, p. 161-167. DOI: 10.1002/ima.22287