AI-Enhanced Test Automation Tool for Desktop Applications
Abstract
Test automation is an essential part of the software testing process. This study aims to develop an AI-enhanced test automation tool for testing the user interfaces of desktop applications. The detection of each object in the graphical user interfaces of the software will be carried out using the object detection capabilities of YOLOv9 and Faster R-CNN models.
The study emphasizes the importance of preprocessing steps for achieving successful outcomes in object detection processes. These preprocessing steps include image resizing, data augmentation techniques, and balancing the dataset. Additionally, the correct selection and optimization of hyperparameters (e.g., learning rate, number of epochs, network depth, and anchor box dimensions) in object detection models play a critical role in improving model performance.
In this study, data analysis techniques using Python were utilized for hyperparameter optimization. Hyperparameters were evaluated and optimized based on metrics such as model accuracy, loss curves, and training time. As a result, high performance was achieved for both the test automation tool and the object detection process. This approach demonstrates the power of artificial intelligence and data analytics in test automation processes, serving as a significant example for both educational and practical applications.
Keywords
Supporting Institution
Havelsan
Ethical Statement
We, the authors of this manuscript titled "[AI-Enhanced-Test-Automation-Tool]," hereby declare that the study was conducted in accordance with ethical standards and guidelines. All participants in the study provided informed consent. Personal data of participants were anonymized to ensure confidentiality.
The authors declare that there are no conflicts of interest regarding the publication of this paper. The funding agency had no involvement in the design, execution, or publication process of this study.All data generated and analyzed during this study are available upon request. The datasets used in this study have been anonymized to protect participant privacy. We affirm that this manuscript is original, has not been published elsewhere, and is not currently under consideration by any other journal. By submitting this manuscript, we confirm that all ethical requirements have been fulfilled and that the research has been conducted responsibly.
Sincerely,
Nagme Cinel Comertler
Ankara University
26.12.2024
Thanks
I acknowledge the invaluable contributions of everyone who helped make this work possible. Any errors or omissions in this work are solely the responsibility of the authors.
References
- [1] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, and G. Li, "Object detection for graphical user interface: old fashioned or deep learning or a combination" in Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020), New York, NY, USA, 2020, pp. 1202-1214. [Online]. https://doi.org/10.1145/3368089.3409691
- [2] Dwivedi, S. K., & Rawat, B. (2015, October). A review paper on data preprocessing: A critical phase in web usage mining process. In 2015 International Conference on Green Computing and Internet of Things (ICGCIoT) (pp. 506-510). IEEE.
- [3] C. Zhang, T. Shi, J. Ai, and W. Tian, "Construction of GUI Elements Recognition Model for AI Testing based on Deep Learning," in Proc. 2021 8th International Conference on Dependable Systems and Their Applications (DSA), Yinchuan, China, 2021, pp. 508-515. doi: 10.1109/DSA52907.2021.00075.
- [4] Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), 1, I-511-I-518.
- [5] Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 1, 886-893.
- [6] Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 32(9), 1627-1645
- [7] Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(6), 1137–1149.
- [8] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. Proceedings of the European Conference on Computer Vision (ECCV 2016), 21–37
Details
Primary Language
English
Subjects
Artificial Intelligence (Other)
Journal Section
Research Article
Authors
Publication Date
May 1, 2025
Submission Date
December 26, 2024
Acceptance Date
April 2, 2025
Published in Issue
Year 2025 Volume: 5 Number: 1
APA
Cinel Cömertler, N. (2025). AI-Enhanced Test Automation Tool for Desktop Applications. Artificial Intelligence Theory and Applications, 5(1), 42-50. https://izlik.org/JA64WR76ND
AMA
1.Cinel Cömertler N. AI-Enhanced Test Automation Tool for Desktop Applications. AITA. 2025;5(1):42-50. https://izlik.org/JA64WR76ND
Chicago
Cinel Cömertler, Nagme. 2025. “AI-Enhanced Test Automation Tool for Desktop Applications”. Artificial Intelligence Theory and Applications 5 (1): 42-50. https://izlik.org/JA64WR76ND.
EndNote
Cinel Cömertler N (May 1, 2025) AI-Enhanced Test Automation Tool for Desktop Applications. Artificial Intelligence Theory and Applications 5 1 42–50.
IEEE
[1]N. Cinel Cömertler, “AI-Enhanced Test Automation Tool for Desktop Applications”, AITA, vol. 5, no. 1, pp. 42–50, May 2025, [Online]. Available: https://izlik.org/JA64WR76ND
ISNAD
Cinel Cömertler, Nagme. “AI-Enhanced Test Automation Tool for Desktop Applications”. Artificial Intelligence Theory and Applications 5/1 (May 1, 2025): 42-50. https://izlik.org/JA64WR76ND.
JAMA
1.Cinel Cömertler N. AI-Enhanced Test Automation Tool for Desktop Applications. AITA. 2025;5:42–50.
MLA
Cinel Cömertler, Nagme. “AI-Enhanced Test Automation Tool for Desktop Applications”. Artificial Intelligence Theory and Applications, vol. 5, no. 1, May 2025, pp. 42-50, https://izlik.org/JA64WR76ND.
Vancouver
1.Nagme Cinel Cömertler. AI-Enhanced Test Automation Tool for Desktop Applications. AITA [Internet]. 2025 May 1;5(1):42-50. Available from: https://izlik.org/JA64WR76ND