Prioritization of Regression Test Cases Based on Machine Learning Methods

Abstract

Due to resource and time constraints involved in the software testing process, it is not possible to implement all test scenarios for each release. Test scenarios can be prioritized according to certain criteria defined by the developers to ensure effective execution of the testing process and detection of errors. This study investigated whether machine learning based models could be used to prioritize test scenarios created in regression testing. It is attempted to determine which tests can be prioritized for execution based on different independent variables. In total, each of the 964 test scenarios in the dataset was labelled as minor (482) and major (482) by two experts. In the models, the number of related requirements, the number of related errors, and the age of the scenario were used as independent variables, and the scenario classes labelled as minor - major were taken as the target variable. The scenarios were pre-processed using natural language processing techniques and different machine learning algorithms were used for model development. In the classification based on test scenarios, the random forest algorithm showed the best performance with a F1-score of 81%. In the classification based on the number of related requirements, the number of interrelated errors, and the age of the test scenarios, the random forest model once again demonstrated the highest success rate at 79%. This study demonstrates that machine learning techniques offer a variety of models for test case prioritization.

Keywords

• Machine learning
• Natural language processing
• Software testing
• Regression testing
• Test case prioritization

References

1. [1] Atifi, M., Mamouni, A., and Marzak, A., “A comparative study of software testing techniques”, Lecture Notes in Computer Science, 10299: 373–390, (2017). DOI: 10.1007/978-3-319-59647-1_27
2. [2] Lonetti, F., and Marchetti, E., “Emerging Software Testing Technologies”, Advances in Computers, 108: 91–143, (2018).
3. [3] Shah, U.S., Jinwala, D.C., and Patel, S.J., “An Excursion to Software Development Life Cycle Models”, ACM SIGSOFT Software Engineering Notes, 41(1): 1–6, (2016). DOI: 10.1145/2853073.2853080
4. [4] Khatibsyarbini, M., Isa, M.A., Jawawi, D.N.A., and Tumeng, R., “Trend Application of Machine Learning in Test Case Prioritization: A Review on Techniques”, IEEE Access, 9: 166262–166282, (2021). DOI: 10.1109/ACCESS.2021.3135508
5. [5] Khatibsyarbini, M., Isa, M.A., Jawawi, D.N.A., and Tumeng, R., “Test case prioritization approaches in regression testing: A systematic literature review”, Information and Software Technology, 93: 74–93, (2018). DOI: 10.1016/j.infsof.2017.08.014
6. [6] Harold, M.J., “Testing: A roadmap”, Proceedings of the Conference on the Future of Software Engineering, ICSE 2000, 61–72, (2000). DOI: 10.1145/336512.336532
7. [7] Lou, Y., Chen, J., Zhang, L., and Hao, D., “A Survey on Regression Test-Case Prioritization”, Advances in Computers, 113: 1–46, (2019).
8. [8] Minhas, N.M., Petersen, K., Börstler, J., and Wnuk, K., “Regression testing for large-scale embedded software development – Exploring the state of practice”, Information and Software Technology, 120: 106254, (2020). DOI: 10.1016/j.infsof.2019.106254

9. [9] Rosero, R.H., Gomez, O.S., and Rodriguez, G., “Regression Testing of Database Applications under an Incremental Software Development Setting”, IEEE Access, 5: 18419–18428, (2017). DOI: 10.1109/ACCESS.2017.2749502
10. [10] Ahmad, S.F., Singh, D.K., and Suman, P., “Prioritization for regression testing using ant colony optimization based on test factors”, Advances in Intelligent Systems and Computing, 624: 1353–1360, (2018). DOI: 10.1007/978-981-10-5903-2_142
11. [11] Boyar, T., Oz, M., Oncu, E., and Aktas, M.S., “A Novel Approach to Test Case Prioritization for Software Regression Tests”, Lecture Notes in Computer Science, 12955: 201–216, (2021). DOI: 10.1007/978-3-030-87007-2_15
12. [12] Li, Z., Harman, M., and Hierons, R.M., “Search algorithms for regression test case prioritization”, IEEE Transactions on Software Engineering, 33(4): 225–237, (2007). DOI: 10.1109/TSE.2007.38
13. [13] Desikan, S., and Ramesh, G., “Developing and Baselining Test Cases”, Software Testing: Principles and Practices, Pearson Education, 376, (2007).
14. [14] Prado Lima, J.A., and Vergilio, S.R., “Test Case Prioritization in Continuous Integration environments: A systematic mapping study”, Information and Software Technology, 121: 106268, (2020). DOI: 10.1016/J.INFSOF.2020.106268
15. [15] Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L., “How does regression test prioritization perform in real-world software evolution?”, Proceedings of the International Conference on Software Engineering, 14-22, 535–546, (2016). DOI: 10.1145/2884781.2884874
16. [16] Rothermel, G., Untcn, R.H., Chu, C., and Harrold, M.J., “Prioritizing test cases for regression testing”, IEEE Transactions on Software Engineering, 27(10): 929–948, (2001). DOI: 10.1109/32.962562
17. [17] Ma, T., Zeng, H., and Wang, X., “Test case prioritization based on requirement correlations”, Proceedings of the 17th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2016, 419–424, (2016). DOI: 10.1109/SNPD.2016.7515934
18. [18] Afshinpour, B., Groz, R., Amini, M.R., Ledru, Y., and Oriat, C., “Reducing regression test suites using the Word2Vec natural language processing tool”, CEUR Workshop Proceedings, 2799: 43–53, (2020).
19. [19] Ledru, Y., Petrenko, A., Boroday, S., and Mandran, N., “Prioritizing test cases with string distances”, Automated Software Engineering, 19(1): 65–95, (2012). DOI: 10.1007/s10515-011-0093-0
20. [20] Thomas, S.W., Hemmati, H., Hassan, A.E., and Blostein, D., “Static test case prioritization using topic models”, Empirical Software Engineering, 19(1): 182–212, (2014). DOI: 10.1007/s10664-012-9219-7
21. [21] Spieker, H., Gotlieb, A., Marijan, D., and Mossige, M., “Reinforcement learning for automatic test case prioritization and selection in continuous integration”, Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, 12–22, (2017). DOI: 10.1145/3092703.3092709
22. [22] Musa, S., Sultan, A.B.M., Ghani, A.A.A., and Baharom, S., “Regression Test Case Selection & Prioritization Using Dependence Graph and Genetic Algorithm”, IOSR Journal of Computer Engineering, 16(3): 38–47, (2014). DOI: 10.9790/0661-16343847
23. [23] Sharif, A., Marijan, D., and Liaaen, M., “DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing”, Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, 525–534, (2021). DOI: 10.1109/ICSME52107.2021.00053
24. [24] Gokce, N., and Eminli, M., “Model-Based Test Case Prioritization Using Neural Network Classification”, Computer Science and Engineering: An International Journal, 4(1): 15–25, (2014). DOI: 10.5121/cseij.2014.4102
25. [25] Nichols, J.A., Herbert Chan, H.W., and Baker, M.A.B., “Machine learning: applications of artificial intelligence to imaging and diagnosis”, Biophysical Reviews, 11(1): 111–118, (2019). DOI: 10.1007/s12551-018-0449-9
26. [26] Meçe, E.K., Paci, H., and Binjaku, K., “The Application of Machine Learning in Test Case Prioritization - A Review”, European Journal of Electrical and Computer Engineering, 4(1): 1-6, (2020). DOI: 10.24018/ejece.2020.4.1.128
27. [27] Blanck, M., “Predicting Price Residuals in Online Car Marketplaces with Natural Language Processing”, Karlsruhe Institute of Technology, (2019).
28. [28] Sidey-Gibbons, J.A.M., and Sidey-Gibbons, C.J., “Machine learning in medicine: a practical introduction”, BMC Medical Research Methodology, 19(1): 1–18, (2019). DOI: 10.1186/s12874-019-0681-4
29. [29] Toraman, C., Yilmaz, E.H., Sahinuc, F., and Ozcelik, O., “Impact of Tokenization on Language Models: An Analysis for Turkish”, ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4): 1-10, (2023). DOI: 10.1145/3578707
30. [30] Korenius, T., Laurikkala, J., Järvelin, K., and Juhola, M., “Stemming and lemmatization in the clustering of finnish text documents”, Proceedings of the International Conference on Information and Knowledge Management, 625–633, (2004). DOI: 10.1145/1031171.1031285
31. [31] Özkan, M., and Kar, G., “Multiclass Classification of Scientific Texts Written in Turkish by Applying Deep Learning Technique”, Journal of Engineering Sciences and Design, 10(2): 504–519, (2022). DOI: 10.21923/jesd.973181
32. [32] Sun, F., Belatreche, A., Coleman, S., McGinnity, T.M., and Li, Y., “Pre-processing online financial text for sentiment classification: A natural language processing approach”, Proceedings of the IEEE/IAFE Conference on Computational Intelligence for Financial Engineering, 122–129, (2014). DOI: 10.1109/CIFEr.2014.6924063
33. [33] Etaiwi, W., and Naymat, G., “The Impact of applying Different Preprocessing Steps on Review Spam Detection”, Procedia Computer Science, 113: 273–279, (2017). DOI: 10.1016/j.procs.2017.08.368
34. [34] Gharatkar, S., Ingle, A., Naik, T., and Save, A., “Review preprocessing using data cleaning and stemming technique”, Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems, ICIIECS 2017, 1–4, (2017). DOI: 10.1109/ICIIECS.2017.8276011
35. [35] Sutar, S., Kumar, R., Pai, S., and Br, S., “Regression Test cases selection using Natural Language Processing”, Proceedings of the International Conference on Intelligent Engineering and Management, ICIEM 2020, 301–305, (2020). DOI: 10.1109/ICIEM48762.2020.9160225
36. [36] Yang, Y., Huang, X., Hao, X., Liu, Z., and Chen, Z., “An Industrial Study of Natural Language Processing Based Test Case Prioritization”, Proceedings of the 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017, 548–549, (2017). DOI: 10.1109/ICST.2017.66
37. [37] Lachmann, R., “Machine Learning-Driven Test Case Prioritization Approaches for Black-Box Software Testing”, Proceedings of the European Test and Telemetry Conference 2018, 300–309, (2018). DOI: 10.5162/ettc2018/12.4
38. [38] Busjaeger, B., and Xie, T., “Learning for test prioritization: An industrial case study”, Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 975–980, (2016). DOI: 10.1145/2950290.2983954
39. [39] Tahvili, S., Hatvani, L., Ramentol, E., Pimentel, R., Afzal, W., and Herrera, F., “A novel methodology to classify test cases using natural language processing and imbalanced learning”, Engineering Applications of Artificial Intelligence, 95: 103878, (2020). DOI: 10.1016/j.engappai.2020.103878
40. [40] Gao, Y., Zhu, Y., and Zhao, Y., “Dealing with imbalanced data for interpretable defect prediction”, Information and Software Technology, 151: 107016, (2022). DOI: 10.1016/j.infsof.2022.107016
41. [41] Maraş, A., and Selçukcan Erol, Ç., “FuzzyCSampling: A Hybrid fuzzy c-means clustering sampling strategy for imbalanced datasets”, Turkish Journal of Electrical Engineering & Computer Sciences, 31(7): 1223–1236, (2023). DOI: 10.55730/1300-0632.4044
42. [42] Bird, S., Klein, E., and Loper, E., Natural Language Processing with Python, O’Reilly Media Inc., (2009).
43. [43] https://github.com/ahmetax/trstop/tree/master. Access date: 16.08.2024
44. [44] https://snowballstem.org/algorithms/turkish/stemmer.html. Access date: 16.08.2024
45. [45] https://pypi.org/project/snowballstemmer/. Access date: 16.08.2024
46. [46] https://scikit-learn.org/stable/.Access date: 16.08.2024
47. [47] https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction. Access date: 16.08.2024
48. [48] Ajorloo, S., Jamarani, A., Kashfi, M., Haghi Kashani, M., and Najafizadeh, A., “A systematic review of machine learning methods in software testing”, Applied Soft Computing, 162: 1-15, (2024). DOI: 10.1016/j.asoc.2024.111805
49. [49] Troussas, C., Virvou, M., Espinosa, K.J., Llaguno, K., and Caro, J., “Sentiment analysis of Facebook statuses using Naive Bayes Classifier for language learning”, Proceedings of the 4th International Conference on Information, Intelligence, Systems and Applications, 198–205, (2013). DOI: 10.1109/IISA.2013.6623713
50. [50] Balaban, M.E., and Kartal, E., Veri Madencilği ve Makine Öğrenmesi Temel Algoritmaları ve R Dili ile Uygulamaları, Second Edition, İstanbul: Çağlayan Kitapevi, (2018).
51. [51] Cheng, Q., Varshney, P.K., and Arora, M.K., “Logistic regression for feature selection and soft classification of remote sensing data”, IEEE Geoscience and Remote Sensing Letters, 3(4): 491–494, (2006). DOI: 10.1109/LGRS.2006.877949.
52. [52] Suthaharan, S., Machine Learning Models and Algorithms for Big Data Classification, 36: 1-20, Boston, MA: Springer US, (2016).
53. [53] Panesar, A., Machine Learning and AI for Healthcare: Big Data for Improved Health Outcomes, Second Edition, Apress Media LLC, (2021).
54. [54] Belgiu, M., and Drăgu, L., “Random forest in remote sensing: A review of applications and future directions”, ISPRS Journal of Photogrammetry and Remote Sensing, 114: 24–31, (2016). DOI: 10.1016/j.isprsjprs.2016.01.011
55. [55] Nilsson, N.J., Yapay Zekâ - Geçmişi ve Geleceği, Boğaziçi Üniversitesi Yayınevi, (2018).
56. [56] Imandoust, S.B., and Bolandraftar, M., “Application of K-Nearest Neighbor (KNN) Approach for Predicting Economic Events: Theoretical Background”, International Journal of Engineering Research and Applications, 3(5): 605–610, (2013).
57. [57] Pan, R., Bagherzadeh, M., Ghaleb, T.A., and Briand, L., “Test case selection and prioritization using machine learning: a systematic literature review”, Empirical Software Engineering, 27(2): 1–43, (2022). DOI: 10.1007/s10664-021-10066-6
58. [58] Roza, E.A.d., Prado Lima, J.A.d., and Vergilio, S.R., “On the use of contextual information for machine learning based test case prioritization in continuous integration development”, Information and Software Technology, 171: 107444, (2024). DOI: 10.1016/j.infsof.2024.107444