Performance Evaluation of Automated Machine Learning (AutoML) Libraries for Naturel Frequency Estimation of Perforated Diaphragm

Abstract

Automated Machine Learning (AutoML) frameworks are efficient tools for predictive model applications. Several machine learning algorithms can integrate them. This study presents a comprehensive evaluation of four AutoML frameworks that are FLAML, H2O, TPOT, and PyCaret to predict the fundamental frequency of perforated Micro-Electromechanical System (MEMS) diaphragm designs using FEM data. The frameworks performance was basically compared based on predictive accuracy and computational efficiency (time). FLAML’s XGBoost_LimitDepth algorithm achieved the highest test R² score of 0.9982, while H2O’s GBM_6 followed closely with an R² of 0.9958. PyCaret’s GradientBoosting yielded an R² of 0.9954 and TPOT’s best pipeline algorithm (named as Best_Pipeline) an R² of 0.9876. FLAML’s XGBoost_LimitDepth algorithm is the time saving tool with a 11.65 ms test time while 1ms computation time makes TPOT’s Best_Pipeline computationally most efficient tool compared to other frameworks. These results demonstrate that AutoML frameworks are potential tools to use for analyses of complex MEMS designs with fast and reliable results.

Keywords

• AutoML
• Performance
• Prediction
• Perforated diaphragm
• Natural frequency

References

1. Aragão, M. V., Afonso, A. G., Ferraz, R. C., Ferreira, R. G., Leite, S. G., de Figueiredo, F. A., & Mafra, S. B. (2025). A practical evaluation of AutoML tools for binary, multiclass, and multilabel classification. Sci Rep 15, 17682. https://doi.org/10.1038/s41598-025-02149-x
2. Arora, M., Sharma, A., Katoch, S., Malviya, M., & Chopra, S. (2021). A state of the art regressor model’s comparison for effort estimation of agile software. In 2021 2nd International Conference on Intelligent Engineering and Management (ICIEM) (pp. 211–215). IEEE. DOI: 10.1109/ICIEM51511.2021.9445345
3. Breiman, L., Friedman, J., Olshen, R.A. & Stone, C. J. (1984). Classification and Regression Trees. New York: Chapman & Hall/CRC. https://doi.org/10.1201/9781315139470
4. Cha, B.-S., Lee, S.-M., Kanashima, T., Okuyama, M., & Tanaka, T. (2011). Influences of perforation ratio in characteristics of capacitive micromachined ultrasonic transducers in air. Sensors and Actuators A: Physical, 171(2), 191–198. https://doi.org/10.1016/j.sna.2011.08.021
5. Chen, T., & Guestrin, C. (2016). XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,785–794). https://doi.org/10.1145/2939672.2939785
6. Chicco, D., Warrens, M. J., & Jurman, G. (2021). The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science. https://doi.org/10.7717/peerj-cs.623
7. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Mach. Learn, 20, 273–297 https://doi.org/10.1023/A:1022627411411
8. Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Trans. Inf. Theory, 13(1),21–27. DOI: 10.1109/TIT.1967.1053964

9. Dong, J., Chen, Y., Yao, B., Zhang, X., & Zeng, N. (2022). A neural network boosting regression model based on XGBoost. Applied Soft Computing, 125, 109067. https://doi.org/10.1016/j.asoc.2022.109067
10. Duda, R., Hart, P.E., & Stork, D.G. (2000). Pattern classification. 2nd Edition, John Wiley & Sons, Hoboken.
11. Faudzi, A. A. M., Sabzehmeidani, Y., & Suzumori, K. (2020). Application of micro-electro-mechanical systems (MEMS) as sensors: A review. Journal of Robotics and Mechatronics, 32(2), 281–288. https://doi.org/10.20965/jrm.2020.p0281
12. Ganji, B. A., Sedaghat, S.B., Roncaglia, A., & Belsito, L. (2018). Design and fabrication of high performance condenser microphone using C-slotted diaphragm. Microsystem Technologies, 24, 3133–3140. https://doi.org/10.1007/s00542-018-3816-3
13. Gou, L., Wang, H., Ding, Q., Liu, Y., Yang, R., Zhang, F., ... & Cao, G. (2024). Investigation of new accelerometer based on capacitive micromachined ultrasonic transducer (CMUT) with ring-perforation membrane. Micromachines, 15(2), 279. https://doi.org/10.3390/mi15020279
14. Hancock, J. T., & Khoshgoftaar, T. M. (2020). CatBoost for big data: An interdisciplinary review. Journal of Big Data, 7:94, 1–45. https://doi.org/10.1186/s40537-020-00369-8
15. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: data mining, ınference, and prediction. New York: Springer. https://doi.org/10.1007/978-0-387-84858-7
16. Hossain, N., Al Mahmud, M. Z., Hossain, A., Rahman, M. K., Islam, M. S., Tasnim, R., & Mobarak, M. H. (2024).Advances of materials science in MEMS applications: A review. Results in Engineering, 22, 102115. https://doi.org/10.1016/j.rineng.2024.102115
17. Hou, S., Liu, Y., & Yang, Q. (2022). Real-Time Prediction of Rock Mass Classification Based on TBM Operation Big Data and Stacking Technique of Ensemble Learning. J. Rock Mech. Geotech. Eng,14(1), 123–143. https://doi.org/10.1016/j.jrmge.2021.05.004
18. Jin, L. (2021). Applications and Prospects of Mems Sensors in Automotive. 2021 International Conference on Intelligent Manufacturing and Industrial Automation (CIMIA 2021),1-6. Guilin: China. DOI 10.1088/1742-6596/1884/1/012010
19. Kavuncuoğlu, E., & Yıldız, F. (2025). Machine learning (ML) algorithms for prediction of low temperature co-fired ceramic (LTCC) diaphragm performance. Microsystem Technologies, 31, 2529–2549. https://doi.org/10.1007/s00542-025-05872-7
20. Kurz-Kim, J.-R., & Loretan, M. (2014). On the properties of the coefficient of determination in regression models with infinite variance variables. Journal of Econometrics, 181(1), 15–24. https://doi.org/10.1016/j.jeconom.2014.02.004
21. LeDell, E., & Poirier, S. (2020). H2O AutoML: Scalable Automatic Machine Learning. 7th ICML Workshop on Automated Machine Learning.https://api.semanticscholar.org/CorpusID:221338558
22. Lee, S.-M., Cha, B.-S., & Okuyama, M. (2011). Viscous Damping Effect on the CMUT Device in Air. Journal of the Korean Physical Society,58(4), 747-755. DOI:10.3938/jkps.58.747
23. McCullagh, P. (1989). Generalized Linear Models. Boca Raton: Routledge. https://doi.org/10.1201/9780203753736
24. McCulloch, W., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys,5, 115-133. https://doi.org/10.1007/BF02478259
25. Natekin, A., & Knoll, A. (2013). Gradient boosting machines, a tutorial. Front. Neurorobot. 7, https://doi.org/10.3389/fnbot.2013.00021
26. Nguyen Van, L., & Lee, G. (2025). Optimizing stacked ensemble machine learning models for accurate wildfire severity mapping. Remote Sensing, 17(5), 854. https://doi.org/10.3390/rs17050854
27. Olson, R., Bartley, N., Urbanowicz, R., & Moore, J. (2016). Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science. Proceedings of the Genetic and Evolutionary Computation Conference, 485-492. https://doi.org/10.1145/2908812.2908918
28. PyCaret. (2026). Retrieved from https://github.com/pycaret/pycaret
29. Rao, K. S., Samyuktha, W., Vardhan, D. V., Naidu, B. G., Kumar, P. A., Sravani, K.G., & Guha, K. (2020). Design and sensitivity analysis of capacitive MEMS pressure sensor for blood pressure measurement. Microsystem Technologies, 26, 2371–2379. https://doi.org/10.1007/s00542-020-04777-x
30. Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Mach. Learn,85, 333–359. https://doi.org/10.1007/s10994-011-5256-5
31. Shishodia, V., Singh, V., & Thampi, S. G. (2024). Implementation of stack-based ensemble technique for classification of glaciers in the Western Himalayan catchments. Physics and Chemistry of the Earth, Parts A/B/C, 136, 103723. https://doi.org/10.1016/j.pce.2024.103723
32. Sun, Y., Feng, G., Georgiou, G., Niver, E., Noe, K., & Chin, K. (2008). Center embossed diaphragm design guidelines and Fabry–Perot diaphragm fiber optic sensor. Microelectronics Journal,39(5), 711-716. https://doi.org/10.1016/j.mejo.2007.12.020
33. Tsoumakas, G., & Katakis, I. (2007). Multi-label classification: An overview. Int. J. Data Warehousing Mining, 3(3),1-13. DOI: 10.4018/jdwm.2007070101
34. Tsoumakas, G; Katakis, I; Vlahavas, I;. (2010). Random k-labelsets for multilabel classification. IEEE Trans. Knowl. Data Eng,23, 1079–1089. DOI: 10.1109/TKDE.2010.164
35. Vadar, P.S, Moharekar, T.T, & Pol, U.R. (2024). Comparative analysis of automated machine learning libraries: PYCARET, H2O, TPOT, AUTO-SKLEARN, and FLAML. International Journal of Scientific Research in Engineering and Management (IJSREM), 8(11), 1–8. DOI: 10.55041/IJSREM39119.
36. Wang, C., Wu, Q., Weimer, M., & Zhu, E. (2019). FLAML: A Fast and Lightweight AutoML Library. Conference on Machine Learning and Systems. https://api.semanticscholar.org/CorpusID:229348714
37. Yıldız, F. (2021). Comparison of Two Different Circular Diaphragm Models with Central Mass for MEMS. Sakarya University Journal of Science, 25(3),619-628. https://doi.org/10.16984/saufenbilder.737982
38. Yıldız, F. (2025). Frequency and sensitivity evaluation of perforated low temperature co-fired ceramic (LTCC) diaphragm. Microsystem Technologies, 31, 2107–2117. https://doi.org/10.1007/s00542-024-05839-0
39. Yıldız, F., & Kavuncuoğlu, E. (2026). Machine learning-driven predictive modeling of natural frequency and displacement in perforated diaphragms for enhanced structural analysis. Journal of Computational Electronics, 25,27. https://doi.org/10.1007/s10825-025-02467-3
40. Yücelbaş, C., & Yücelbaş, Ş. (2025). A robust approach to type-2 diabetes diagnosis: Combining class imbalance mitigation, elastic-net, and boosting algorithms. Iranian Journal of Science and Technology, Transactions of Electrical Engineering, https://doi.org/10.1007/s40998-025-00992-2
41. Zeng, T., Wu, L., Peduto, D., Glade, T., Hayakawa, Y.S., & Yin, K. (2023). Ensemble learning framework for landslide susceptibility mapping: Different basic classifier and ensemble strategy. Geoscience Frontiers, 14(6), 101645. https://doi.org/10.1016/j.gsf.2023.101645
42. Zhang, M., & Zhou, Z. (2006). Multilabel neural networks with applications to functional genomics and text categorization. IEEE Trans. Knowl. Data Eng, 18(10), 1338–1351. https://doi.org/10.1109/TKDE.2006.162
43. Zhang, Z., Gui, X., Hu, Q., Yang, L., Yang, R., Huang, B., ... & Tang, Z. (2021). Highly sensitive capacitive pressure sensor based on a micropyramid array for health and motion monitoring. Advanced Electronic Materials, 7, 2100174. https://doi.org/10.1002/aelm.202100174
44. Zhu, J., Wang, M., Chen, L., Ni, X., & Ni, H. (2017). An optical fiber Fabry–Perot pressure sensor using corrugated diaphragm and angle polished fiber. Optical Fiber Technology, 34, 42–46. https://doi.org/10.1016/j.yofte.2016.12.004
45. Zhu, L., Lu, W., Luo, C., Xu, Y., & Wang, Z. (2024). An ensemble optimizer with a stacking ensemble surrogate model for identification of groundwater contamination source. Journal of Contaminant Hydrology, 267, 104437. https://doi.org/10.1016/j.jconhyd.2024.104437