An Introduction to Software Testing Methodologies

Abstract

It seems to the authors that ‘testing’ for many of the Computer Engineering or Software Engineering professionals or to laymen means the same as ‘code testing’ or ‘program testing’. Yet, recalling the SDLC (Software Development Life Cycle) steps such as Planning, Analysis, Design, Implementation or Construction and Maintenance, obviously will tell us that coding is just a piece of SDLC included in the Implementation. Therefore, unit testing, integration testing, black box texting or acceptance testing etc. would not cover the whole SDLC. One then needs any methodology, if available, to encompass all steps of SDLC.
The major objective of this article has been to investigate the available methodologies applicable for software testing. During our research, we met TMMi (Test Maturity Model integration) and PRISMA (Product Risk Management) methodologies available in the professional arena. After understanding the characteristics of these methodologies, we attempted to propose a new methodology as a synthesis of these two methodologies.
The details of these two methodologies are given in this article, while a newly proposed methodology described in the article will be applied on a web based problem to be reported later.

Keywords

• PRISMA
• SDLC
• SW Testing Methodology
• TMMi

References

1. Aktaş, A. Z. (1987). Structured Analysis and Design of Information Systems, Prentice-Hall, USA.
2. Alam, M., & Khan, A. I. (2013). Risk-Based Testing Techniques: A Perspective Study. International Journal of Computer Applications, 65(1), 33-41.
3. Amland, S. (2000). Risk-Based Testing: Risk Analysis Fundamentals and Metrics for Software Testing Including a Financial Application Case Study. Journal of Systems and Software, 53(3), 287-295. doi:10.1016/S0164-1212(00)00019-4
4. Bach, J. (1997). Good Enough Quality: Beyond the Buzzword. IEEE Computer, 30(8), 96-98. doi:10.1109/2.607108
5. Bach, J. (1998). A Framework for Good Enough Testing. IEEE Computer, 31(10), 124-126. doi:10.1109/2.722304
6. Bach, J. (1999). Risk-Based Testing: How to conduct heuristic risk analysis. Software Testing and Quality Engineering Magazine, November/December, 23-28. www.satisfice.com/articles/hrbt.pdf
7. Baker, C. L. (1957). Review of “Digital Computer Programming, by D.D. McCracken”. Mathematical Tables and Other Aids to Computation, 11(60), 298-305. doi:10.2307/2001950
8. Black, R. (2011). Advanced Software Testing, Vol.1, Rocky Nook, USA.

9. Boehm, B. W. (1989). Tutorial: Software Risk Management, IEEE Computer Society Press, New York.
10. Broadleaf (2012). A Simple Guide to Risk and Its Management. Broadleaf Capital International Pty Ltd. (Accessed:21/02/2017) broadleaf.com.au/resource-material/a-simple-guide-to-risk-and-its-management/
11. Capgemini, Sogeti & HP (2015). World Quality Report 2015-16 Seventh Edition.
12. Charette, R. N. (1989). Software Engineering Risk Analysis and Management. McGraw-Hill, New York.
13. Ciocoiu, C. N., & Dobrea, R. C. (2010). The Role of Standardization in Improving the Effectiveness of Integrated Risk Management. In: Nota, G. (Eds.) Advances in Risk Management (pp. 1-18). IntechOpen. doi:10.5772/9893
14. CMMI (2010). CMMI for Development, Version 1.3. (Technical Report: CMU/SEI-2010-TR-033). CMMI Product Team, Software Engineering Institute, Carnegie Mellon University. doi:10.1184/R1/6572342.v1
15. Erdogan, G., Y. Li, Y., Runde, R. K., Seehusen, F., & Stølen, K. (2014). Approaches for the combined use of risk analysis and testing: a systematic literature review. International Journal on Software Tools for Technology Transfer, 16(5), 627-642. doi:10.1007/s10009-014-0330-5
16. Felderer, M., & Ramler, R. (2014). Integrating risk-based testing in industrial test processes. Software Quality Journal, 22(3), 543-575. doi:10.1007/s11219-013-9226-y
17. Felderer, M., & Schieferdecker, I. (2014). A taxonomy of risk-based testing. International Journal on Software Tools for Technology Transfer, 16(5), 559-568. doi:10.1007/s10009-014-0332-3
18. Felderer, M., Haisjackl, C., Pekar, V., & Breu, R. (2014). A Risk Assessment Framework for Software Testing. In: Margaria, T., & Steffen, B. (Eds.), Leveraging Applications of Formal Methods, Verification and Validation. Specialized Techniques and Applications. Proceedings of the 6th International Symposium, ISoLA 2014, Part II, (pp. 292-308), October 8-11, Corfu, Greece. doi:10.1007/978-3-662-45231-8_21
19. Felderer, M., Haisjackl, C., Pekar, V., & Breu, R. (2015). An Exploratory Study on Risk Estimation in Risk-Based Testing Approaches. In: Winkler, D., Biffl, S., & Bergsmann, J. (Eds.), Software Quality. Software and Systems Quality in Distributed and Mobile Environments. Proceedings of the 7th International Conference, SWQD 2015, (pp. 20-23), January 20-23, Vienna, Austria. doi:10.1007/978-3-319-13251-8_3
20. Gelperin, D., & Hetzel, B. (1988). The Growth of Software Testing. Communications of the ACM, 31(6), 687-695. doi:10.1145/62959.62965
21. Graham, D., van Veenendaal, E., Evans, I., & Black, R. (2008). Foundations of Software Testing ISTQB Certification. Cengage Learning Emea.
22. Hrabovská, K., Rossi, B., & Pitner, T. (2019). Software Testing Process Models Benefits & Drawbacks: a Systematic Literature Review. arXiv:1901.01450. arxiv.org/abs/1901.01450
23. ISO (2009a). ISO 31000, Principles and Guidelines on Risk Management.
24. ISO (2009b.) ISO/IEC Guide 73:2009, Risk Management -Vocabulary.
25. Karolak, D. W. (1995). Software Engineering Risk Management. Wiley-IEEE Computer Society Press.
26. Laksono, M. A., Budiardjo, E. K., & Ferdinansyah, A. (2019). Assessment of Test Maturity Model: A Comparative Study for Process Improvement. In: Proceedings of the 2nd International Conference on Software Engineering and Information Management, ICSIM 2019, (pp. 110-118), January 10-13, Bali, Indonesia.
27. Liu, H., Kuo, F-C., & Chen, T. Y. (2011a). Comparison of adaptive random testing and random testing under various testing and debugging scenarios. Software: Practice and Experience, 42(8), 1055-1074. doi:10.1002/spe.1113
28. Liu, H., Xie, X., Yang, J., Lu, Y., & Chen, T. Y. (2011b). Adaptive Random Testing Through Test Profiles. Software: Practice and Experience, 41(10), 1131-1154. doi:10.1002/spe.1067
29. Patrício, C., Pinto, R., & Marques, G. (2021) A Study on Software Testing Standard Using ISO/IEC/IEEE 29119-2: 2013. In: Al-Emran M., Shaalan K., & Hassanien A. (Eds) Recent Advances in Intelligent Systems and Smart Applications (pp. 43-62). doi:10.1007/978-3-030-47411-9_3
30. RTI (2002). Planning Report 02-3, The Economic Impacts of Inadequate Infrastructure for Software Testing. Prepared by Research Triangle Institute (RTI) for National Institute of Standards & Technology (NIST). (7007.011) (Accessed:20/02/2017) www.nist.gov/director/planning/upload/report02-3.pdf
31. Roy, G. G. (2004). A risk management framework for software engineering practice. In: Strooper, P. (Eds.) Proceedings of the Australian Software Engineering Conference, ASWEC 2004, (pp. 60-67), April 13-16, Melbourne, Australia. doi:10.1109/ASWEC.2004.1290458
32. Serdaroğlu, D. (2015). Yazılım Test Süreci ve TMMi Modelinde Prisma Yaklaşımı Uygulaması. MSc. Thesis (in Turkish), Başkent Üniversitesi, Bilgisayar Mühendisliği Bölümü.
33. Standards Australia/New Zealand (2004). Risk Management Guidelines Companion to AS/NZS 4360:2004 (HB 436:2004)
34. Stern, R., & Arias, J. C. (2011). Review of Risk Management Methods. Business Intelligence Journal, 4(1), 59-78.
35. van Veenendaal, E. (2011). Practical Risk-Based Testing Product Risk Management: The PRISMA Method. EuroSTAR 2011, Novermber 21-24, Manchester, UK. (Accessed:08/03/2017) www.erikvanveenendaal.nl/NL/files/e-book%20PRISMA.pdf
36. van Veenendaal, E. (2012a). Test Maturity Model integration (TMMi) Release 1.0. TMMi Foundation.
37. van Veenendaal, E. (2012b). Practical Risk-Based Testing - The PRISMA Approach. UTN Publishers.
38. van Veenendaal, E. (2014). Test Process Improvement and Agile: Friends or Foes?. Testing Experience, 27, 24-26. (Accessed:07/03/2017) www.erikvanveenendaal.nl/NL/files/testingexperience27_09_14_van_Veenendaal.pdf
39. van Veenendaal, E., & Cannegieter, J. J. (2013). Test Maturity Model integration (TMMi) Results of the first TMMi benchmark - where are we today?. EuroSTAR. (Accessed:07/03/2017) www.erikvanveenendaal.nl/NL/files/e-book%20TMMi.pdf
40. Wendland, M-F., Kranz, M. & Schieferdecker, I. (2012). A Systematic Approach to Risk-Based Testing Using Risk-annotated Requirements Models. In: The Seventh International Conference on Software Engineering Advances, ICSEA 2012 (pp. 636-642), Lisbon.