AI-Enhanced Test Automation Tool for Desktop Applications

Year 2025, Volume: 5 Issue: 1, 42 - 50, 01.05.2025

Nagme Cinel Cömertler

Abstract

Test automation is an essential part of the software testing process. This study aims to develop an AI-enhanced test automation tool for testing the user interfaces of desktop applications. The detection of each object in the graphical user interfaces of the software will be carried out using the object detection capabilities of YOLOv9 and Faster R-CNN models.
The study emphasizes the importance of preprocessing steps for achieving successful outcomes in object detection processes. These preprocessing steps include image resizing, data augmentation techniques, and balancing the dataset. Additionally, the correct selection and optimization of hyperparameters (e.g., learning rate, number of epochs, network depth, and anchor box dimensions) in object detection models play a critical role in improving model performance.
In this study, data analysis techniques using Python were utilized for hyperparameter optimization. Hyperparameters were evaluated and optimized based on metrics such as model accuracy, loss curves, and training time. As a result, high performance was achieved for both the test automation tool and the object detection process. This approach demonstrates the power of artificial intelligence and data analytics in test automation processes, serving as a significant example for both educational and practical applications.

Keywords

Object Detection, Data Analysis, Test Automation, Deep Learning, Image Classification

Ethical Statement

We, the authors of this manuscript titled "[AI-Enhanced-Test-Automation-Tool]," hereby declare that the study was conducted in accordance with ethical standards and guidelines. All participants in the study provided informed consent. Personal data of participants were anonymized to ensure confidentiality. The authors declare that there are no conflicts of interest regarding the publication of this paper. The funding agency had no involvement in the design, execution, or publication process of this study.All data generated and analyzed during this study are available upon request. The datasets used in this study have been anonymized to protect participant privacy. We affirm that this manuscript is original, has not been published elsewhere, and is not currently under consideration by any other journal. By submitting this manuscript, we confirm that all ethical requirements have been fulfilled and that the research has been conducted responsibly. Sincerely, Nagme Cinel Comertler Ankara University 26.12.2024

Supporting Institution

Havelsan

Thanks

I acknowledge the invaluable contributions of everyone who helped make this work possible. Any errors or omissions in this work are solely the responsibility of the authors.

References

• [1] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, and G. Li, "Object detection for graphical user interface: old fashioned or deep learning or a combination" in Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020), New York, NY, USA, 2020, pp. 1202-1214. [Online]. https://doi.org/10.1145/3368089.3409691
• [2] Dwivedi, S. K., & Rawat, B. (2015, October). A review paper on data preprocessing: A critical phase in web usage mining process. In 2015 International Conference on Green Computing and Internet of Things (ICGCIoT) (pp. 506-510). IEEE.
• [3] C. Zhang, T. Shi, J. Ai, and W. Tian, "Construction of GUI Elements Recognition Model for AI Testing based on Deep Learning," in Proc. 2021 8th International Conference on Dependable Systems and Their Applications (DSA), Yinchuan, China, 2021, pp. 508-515. doi: 10.1109/DSA52907.2021.00075.
• [4] Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), 1, I-511-I-518.
• [5] Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 1, 886-893.
• [6] Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 32(9), 1627-1645
• [7] Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(6), 1137–1149.
• [8] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. Proceedings of the European Conference on Computer Vision (ECCV 2016), 21–37
• [9] Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934
• [10] Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), 580–587.
• [11] Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(60), 1–48.
• [12] Little, R. J., & Rubin, D. B. (2019). Statistical analysis with missing data (Vol. 793). John Wiley & Sons.
• [13] Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281–305.
• [14] G. Li, G. Baechler, M. Tragut, and Y. Li, "Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale," in Proc. of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22), New York, NY, USA, 2022, Article 67, pp. 1-13. [Online]. https://doi.org/10.1145/3491102.3502042
• [15] Huang, Jonathan et al. “Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 3296-3297.