Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data

Zafer Yavuz; Turgay Tugay Bilgin

doi:10.30939/ijastech..1676422

Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data

Abstract

Considering the high data production in automotive sector production lines, the analysis of this data is of critical importance for predictive maintenance, energy efficiency, and quality control processes. However, increasing data volume challenges the limits of traditional methods and requires consideration of the performance evaluation of different libraries. This paper aims to compare the performance characteristics of Pandas, Dask, Modin, Vaex and Polars libraries in the Python ecosystem for processing large datasets obtained from welding machines used in modern automotive production systems. The study utilized real production data from Matay, an automotive parts supplier, consisting of approximately 30 days of exhaust production machine data with a size of 17 GB containing 106,167,826 rows. Subsets of different sizes (10K, 100K, 1M, 10M rows) were created from this dataset, and 11 different experiments were conducted on selected columns. These experiments cover the topics of reading data, filtering, sorting, grouping, merging, writing data in different formats (csv, parquet) and handling missing data. Then the experiments were evaluated based on three different metrics: total execution time, total memory usage, and CPU execution time. Each experiment was repeated 3 times and average values were recorded. In conclusion, this study demonstrates that Polars may be more advantageous for performance-oriented applications across all data scales. Ultimately, the strategic selection of these data processing tools serves as a critical enabler for digital transformation in the automotive industry; thereby facilitating the integration of digital twins and AI-driven quality control into high-performance Industry 4.0 ecosystems.

Keywords

References

[1] Temiz RÖ, Onan M, Cebi H, Aslanlar S, Talaş Ş. Effect of Elec-trode Type and Weld Current on Service Life of Resistance Spot Weld Electrode. International Jorunal of Automotive Science and Technology. 2024;8(1):52-64. https://doi.org/10.30939/ijastech..1315759.
[2] Çıkmak S. Pythagorean Fuzzy AHP Approach for Evaluating the Importance Level of Industry 4.0 Technologies in the Automotive Manufacturing Industry. International Jorunal of Automotive Sci-ence and Technology. 2025;9(1):26-39. https://doi.org/10.30939/ijastech..1522257.
[3] Blachowicz T, Wylezek J, Sokol Z, Bondel M. Real-Time Analy-sis of Industrial Data Using the Unsupervised Hierarchical Densi-ty-Based Spatial Clustering of Applications with Noise Method in Monitoring the Welding Process in a Robotic Cell. Information. 2025; 16(2):79. https://doi.org/10.3390/info16020079.
[4] Shanbhag S, Chimalakonda S. An Exploratory Study on Energy Consumption of Dataframe Processing Libraries. In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE; 2023. p. 284–95. https://doi.org/10.1109/MSR59073.2023.00048.
[5] Abeykoon V, Charles Fox G. Trends in High-Performance Data Engineering for Data Analytics. In: New Trends and Challenges in Open Data. IntechOpen; 2023. https://doi.org/10.5772/intechopen.1001458.
[6] Castro O, Bruneau P, Sottet JS, Torregrossa D. Landscape of High-Performance Python to Develop Data Science and Machine Learning Applications. ACM Computing Surveys. 2024 Mar 31;56(3):1–30. https://doi.org/10.1145/3617588.
[7] Chen Y, Su S, Yang H. Convolutional Neural Network Analysis of Recurrence Plots for Anomaly Detection. International Journal of Bifurcation and Chaos. 2020 Jan 12;30(01):2050002. https://doi.org/10.1142/s0218127420500029.
[8] Wang B, Li Y, Luo Y, Li X, Freiheit T. Early event detection in a deep-learning driven quality prediction model for ultrasonic weld-ing. Journal of Manufacturing Systems. 2021 Jul; 60:325–36. https://doi.org/10.1016/j.jmsy.2021.06.009.

[9] Kim YM, Yoon IU, Myung H, Kim JH. Bi-Directional Convolu-tional Recurrent Reconstructive Network for Welding Defect De-tection. IEEE Access. 2021; 9:135316–25. https://doi.org/10.1109/access.2021.3116799.
[10] Yang G, Zhong Y, Yang L, Li J, Liang Y, Du R. Anomaly De-tection of Automobile Welding Workstation Based on Stacked Autoencoders and Long Short-Term Memory Network. In: 2021 China Automation Congress (CAC). IEEE; 2021. p. 1433–7. https://doi.org/10.1109/cac53003.2021.9727326.
[11] Nele L, Mattera G, Vozza M. Deep Neural Networks for Defects Detection in Gas Metal Arc Welding. Applied Sciences. 2022 Apr 2;12(7):3615. https://doi.org/10.3390/app12073615.
[12] Meyer K, Mahalec V. Anomaly detection methods for infrequent failures in resistive steel welding. Journal of Manufacturing Pro-cesses. 2022 Mar; 75:497–513. https://doi.org/10.1016/j.jmapro.2021.12.003.
[13] Naddaf Shargh S, Naddaf-Sh M, Dalton M, Ramezani S, Kashani AR, Zargarzadeh H. Explainable Models for Multivariate Time-series Defect Classification of Arc Stud Welding. Interna-tional Journal of Prognostics and Health Management. 2023 Feb 13;14(3). https://doi.org/10.36001/ijphm.2023.v14i3.3125.
[14] Russell M, Kershaw J, Xia Y, Lv T, Li Y, Ghassemi-Armaki H, et al. Comparison and explanation of data-driven modeling for weld quality prediction in resistance spot welding. Journal of In-telligent Manufacturing. 2024 Mar 1;35(3):1305–19. https://doi.org/10.1007/s10845-023-02108-1.
[15] Liu L, Chen H, Chen S. Quality analysis of CMT lap welding based on welding electronic parameters and welding sound. Jour-nal of Manufacturing Processes. 2022 Feb; 74:1–13. https://doi.org/10.1016/j.jmapro.2021.11.055.
[16] Chang F, Zhou G, Ding K, Li J, Jing Y, Hui J, et al. A CNN-LSTM and Attention-Mechanism-Based Resistance Spot Welding Quality Online Detection Method for Automotive Bodies. Math-ematics. 2023 Nov 7;11(22):4570. https://doi.org/10.3390/math11224570.
[17] Zhou X, Chen M, Shang W, Shen H, Xu H. Real time Monitor-ing Method of Welding Defects Based on LSTM Sequence Mod-el. In: 2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA). IEEE; 2023. p. 1203–7. https://doi.org/10.1109/ICIBA56860.2023.10165203.
[18] Peng P, Fan K, Fan X, Zhou H, Guo Z. Real-Time Defect De-tection Scheme Based on Deep Learning for Laser Welding Sys-tem. IEEE Sensors Journal. 2023 Aug 1;23(15):17301–9. https://doi.org/10.1109/jsen.2023.3277732.
[19] Yan S, Chen B, Tan C, Song X, Wang G. A data-driven time-sequence feature-based composite network of time-distributed CNN-LSTM for detecting pore defects in laser penetration weld-ing. Journal of Intelligent Manufacturing. 2024 May 25; https://doi.org/10.1007/s10845-024-02391-6.
[20] Chen H, Dong X, Kong Y, Chen Z, Zheng S, Hu X, et al. Online prediction of mechanical and electrical quality in ultrasonic metal welding using time series generation and deep learning. En-gineering Failure Analysis. 2024 Jun; 160:108162. https://doi.org/10.1016/j.engfailanal.2024.108162.
[21] Asif K, Zhang L, Derrible S, Indacochea JE, Ozevin D, Ziebart B. Machine learning model to predict welding quality using air-coupled acoustic emission and weld inputs. Journal of Intelligent Manufacturing. 2022 Mar 27;33(3):881–95. https://doi.org/10.1007/s10845-020-01667-x.
[22] Liu Y, Yuan K, Li T, Li S, Ren Y. NDT Method for Line Laser Welding Based on Deep Learning and One-Dimensional Time-Series Data. Applied Sciences. 2022 Aug 4;12(15):7837. https://doi.org/10.3390/app12157837.
[23] Cho HW, Shin SJ, Seo GJ, Kim DB, Lee DH. Real-time anoma-ly detection using convolutional neural network in wire arc addi-tive manufacturing: Molybdenum material. Journal of Materials Processing Technology. 2022 Apr; 302:117495. https://doi.org/10.1016/j.jmatprotec.2022.117495.
[24] Choi JS, Choi SW, Lee EB. Modeling of Predictive Maintenance Systems for Laser-Welders in Continuous Galvanizing Lines Based on Machine Learning with Welder Control Data. Sustaina-bility. 2023 May 7;15(9):7676. https://doi.org/10.3390/su15097676.
[25] Widanage C, Perera N, Abeykoon V, Kamburugamuve S, Kanewala TA, Maithree H, et al. High Performance Data Engi-neering Everywhere. In: 2020 IEEE International Conference on Smart Data Services (SMDS). IEEE; 2020. p. 122–32. https://doi.org/10.1109/smds49396.2020.00022.
[26] Sinthong P, Carey MJ. AFrame: Extending DataFrames for Large-Scale Modern Data Analysis. In: 2019 IEEE International Conference on Big Data (Big Data). IEEE; 2019. p. 359–71. https://doi.org/10.1109/bigdata47090.2019.9006303.
[27] Sinthong P, Carey MJ. PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes. In: Proceedings of the VLDB Endowment. 2021 Jul 1;14(11):2296–304. https://doi.org/10.14778/3476249.3476281.
[28] Petersohn D, Macke S, Xin D, Ma W, Lee D, Mo X, et al. To-wards Scalable Dataframe Systems. 2020 Jan 3; https://doi.org/10.48550/arXiv.2001.00888.
[29] Petersohn D, Tang D, Durrani R, Melik-Adamyan A, Gonzalez JE, Joseph AD, et al. Flexible rule-based decomposition and metadata independence in modin. Proceedings of the VLDB En-dowment. 2021 Nov 4;15(3):739–51. https://doi.org/10.14778/3494124.3494152.
[30] Xin D, Petersohn D, Tang D, Wu Y, Gonzalez JE, Hellerstein JM, et al. Enhancing the interactivity of dataframe queries by lev-eraging think time. Bulletin of the IEEE Computer Society Tech-nical Committee on Data Engineering. 2021; 44(1):66–78. https://doi.org/10.48550/arXiv.2103.02145.
[31] Emani V, Floratou A, Curino C. PyFroid: Scaling Data Analysis on a Commodity Workstation. In: Proceedings of the International Conference on Extending Database Technology (EDBT). 2024. https://doi.org/10.48786/edbt.2024.06.
[32] Jindal A, Venkatesh Emani K, Daum M, Poppe O, Haynes B, Pavlenko A, et al. Magpie: Python at Speed and Scale using Cloud Backends. In: 11th Annual Conference on Innovative Data Systems Research (CIDR ’21). Chaminade, USA; 2021. https://api.semanticscholar.org/CorpusID:231782138.
[33] Kläbe S, Hagedorn S. When Bears get Machine Support: Apply-ing Machine Learning Models to Scalable DataFrames with Griz-zly. In: Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI). Gesellschaft fur Informat-ik (GI); 2021. p. 195–214. https://doi.org/10.18420/btw2021-10.
[34] Pereira R, Couto M, Ribeiro F, Rua R, Cunha J, Fernandes JP, et al. Ranking programming languages by energy efficiency. Sci-ence of Computer Programming. 2021 May; 205:102609. https://doi.org/10.1016/j.scico.2021.102609.
[35] Georgiou S, Kechagia M, Sharma T, Sarro F, Zou Y. Green AI: Do deep learning frameworks have different costs? In: Proceed-ings of the 44th International Conference on Software Engineer-ing. 2022. https://doi.org/10.1145/3510003.3510221.
[36] Reya NF, Ahmed A, Zaman T, Islam MdM. GreenPY: Evaluat-ing Application-Level Energy Efficiency in Python for Green Computing. Annals of Emerging Technologies in Computing. 2023 Jul 1;7(3):92–110. https://doi.org/10.33166/aetic.2023.03.005.
[37] Nahrstedt F, Karmouche M, Bargieł K, Banijamali P, Nalini Pradeep Kumar A, Malavolta I. An Empirical Study on the Ener-gy Usage and Performance of Pandas and Polars Data Analysis Python Libraries. In: Proceedings of the 28th International Con-ference on Evaluation and Assessment in Software Engineering. New York, NY, USA: ACM; 2024. p. 58–68. https://doi.org/10.1145/3661167.3661203.
[38] Stancin I, Jovic A. An overview and comparison of free Python libraries for data mining and big data analysis. In: 2019 42nd In-ternational Convention on Information and Communication Tech-nology, Electronics and Microelectronics (MIPRO). IEEE; 2019. p. 977–82. https://doi.org/10.23919/mipro.2019.8757088.
[39] Shovon AR, Dyken LR, Green O, Gilray T, Kumar S. Acceler-ating Datalog applications with cuDF. 2022 IEEE/ACM Work-shop on Irregular Applications: Architectures and Algorithms (IA3) [Internet]. 2022 Nov: 41–45. https://doi.org/10.1109/IA356718.2022.00012.
[40] Moreno M, Vilaça R, Ferreira PG. Scalable transcriptomics analysis with Dask: applications in data science and machine learning. BMC Bioinformatics. 2022 Nov 30;23(1):514. https://doi.org/10.1186/s12859-022-05065-3.
[41] Mozzillo A, Zecchini L, Gagliardelli L, Aslam A, Bergamaschi S, Simonini G. Evaluation of Dataframe Libraries for Data Prepara-tion on a Single Machine. 2023. https://doi.org/10.48550/arXiv.2312.11122.
[42] Psallidas F, Zhu Y, Karlas B, Henkel J, Interlandi M, Krishnan S, et al. Data science through the looking glass. ACM SIGMOD Record. 2022 Jul 29;51(2):30–7. https://doi.org/10.1145/3552490.3552496.
[43] Rocklin M. Dask: Parallel Computation with Blocked algorithms and Task Scheduling. In: Proceedings of the Python in Science Conferences. 2015 Jan 1;126–32. https://doi.org/10.25080/majora-7b98e3ed-013.
[44] Ge R, Feng X, Song S, Chang HC, Li D, Cameron KW. Pow-erPack: Energy Profiling and Analysis of High-Performance Sys-tems and Applications. IEEE Transactions on Parallel and Dis-tributed Systems. 2009 May 4;21(5):658–71. https://doi.org/10.1109/tpds.2009.76.

Details

Primary Language

English

Subjects

Automotive Engineering (Other)

Journal Section

Research Article

Authors

Zafer Yavuz ^*
0000-0002-8427-066X
Türkiye

Turgay Tugay Bilgin
0000-0002-9245-5728
Türkiye

Publication Date

February 11, 2026

Submission Date

April 15, 2025

Acceptance Date

January 20, 2026

Published in Issue

Year 2026 Volume: 10 Number: 1

DOI

https://doi.org/10.30939/ijastech..1676422

IZ

https://izlik.org/JA32ZX88XZ

Cite

RIS / Bibtex

APA

Yavuz, Z., & Bilgin, T. T. (2026). Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data. International Journal of Automotive Science And Technology, 10(1), 26-39. https://doi.org/10.30939/ijastech..1676422

AMA

1.Yavuz Z, Bilgin TT. Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data. IJASTECH. 2026;10(1):26-39. doi:10.30939/ijastech.1676422

Chicago

Yavuz, Zafer, and Turgay Tugay Bilgin. 2026. “Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data”. International Journal of Automotive Science And Technology 10 (1): 26-39. https://doi.org/10.30939/ijastech. 1676422.

EndNote

Yavuz Z, Bilgin TT (February 1, 2026) Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data. International Journal of Automotive Science And Technology 10 1 26–39.

IEEE

[1]Z. Yavuz and T. T. Bilgin, “Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data”, IJASTECH, vol. 10, no. 1, pp. 26–39, Feb. 2026, doi: 10.30939/ijastech..1676422.

ISNAD

Yavuz, Zafer - Bilgin, Turgay Tugay. “Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data”. International Journal of Automotive Science And Technology 10/1 (February 1, 2026): 26-39. https://doi.org/10.30939/ijastech. 1676422.

JAMA

1.Yavuz Z, Bilgin TT. Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data. IJASTECH. 2026;10:26–39.

MLA

Yavuz, Zafer, and Turgay Tugay Bilgin. “Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data”. International Journal of Automotive Science And Technology, vol. 10, no. 1, Feb. 2026, pp. 26-39, doi:10.30939/ijastech. 1676422.

Vancouver

1.Zafer Yavuz, Turgay Tugay Bilgin. Optimizing Large-Scale Data Processing in Smart Manufacturing: A Benchmarking Study on Automotive Industry Data. IJASTECH. 2026 Feb. 1;10(1):26-39. doi:10.30939/ijastech. 1676422