FACE VALIDATION IN LARGE-SCALE SIMULATION PROJECTS

Abstract

Face validation is an informal type of validation, usually applied in the early stages of a simulation project or model development process. Face validation is used to determine the reasonableness of a model or simulation when advanced validation methods such as statistical analysis or sensitivity analysis are not applicable. In large-scale simulation projects such as warfare simulation applications, which consist of a large number and variety of models that cannot be evaluated independently and contain high levels of uncertainty, face validation should be utilized, but no standard method for face validation can be proposed. In this paper, we review the literature and practical applications of face validation. We propose a process to improve the effectiveness of face validation. The proposed process is expected to help to realize a measurable, concrete and practical face validation, especially in large-scale projects involving high-fidelity complex models and constructive simulations.

Keywords

• Verification
• Validation
• Face Validation
• Simulation Model

Supporting Institution

TÜBİTAK SAVTAG

Project Number

116A003

Thanks

This paper and the research behind it would not have been possible without the support of TÜBİTAK SAVTAG.

References

1. Greenwood AG, "A Specification for Effective Simulation Project Management", 2020 Winter Simulation Conference (WSC), pp. 2482-2492, doi: 10.1109/WSC48552.2020.9383982, 2020.
2. Balci O, "Verification, validation, and accreditation'', In Proceedings of the 30th conference on Winter simulation (WSC '98), IEEE Computer Society Press, Washington, DC, USA, 41–4, 1998.
3. Kaya MC, Karamanlioglu A, Çetintaş Ç, Çilden E, Canberi H, Oğuztüzün H, “A configurable gateway for DDS-HLA interoperability”, Simulation Series, vol. 2019-July, 2019.
4. Hocaoğlu MF, Kolsuz H, Reşber E, “Simulation Conceptual Model And Scenario Verification Based On Adsif”, USMOS 2015- National Defense Applications Modeling Simulation Conference, Ankara, 2015.
5. Balci O, "Verification, Validation, and Testing of Models", In: Gass S.I., Fu M.C. (eds) Encyclopedia of Operations Research and Management Science. Springer, Boston, MA.
6. Öztürk V, Arar ÖF, Rende FŞ, Öztemel E, Sezer S, “Validation of railway vehicle dynamic models in training simulators”, Veh. Syst. Dyn., vol. 55, no. 1, pp. 41–71,
7. Haklidir M, Guven AF, Eroglu O, Aldogan D, Tasdelen I, "High fidelity modeling and simulation of submarine in a commercial computer generated forces toolkit", In Proceedings of the 2009 Summer Computer Simulation Conference (SCSC '09). Society for Modeling & Simulation International, Vista, CA, 299–306, 2009.
8. Senyurek L, Koksal S, Genc HM, Aldogan D, Haklidir M, “Implementation Of Fuzzy Control For Surface Platforms In A Computer Generated Forces Toolkit”, International Conference on Computational Intelligence for Modelling, Control and Automation (CIMCA08), Vienna, Austria, 2008.

9. Kalra N, Paddock SM, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?”, Transp. Res. Part A Policy Pract. vol. 94, pp. 182–193, doi: 10.1016/j.tra.2016.09.010, 2016.
10. Bendre HH, Rajender A, Barbosa PV, Wason SEL, “Robotic dismembered pyeloplasty surgical simulation using a 3D-printed silicone-based model: development, face validation and crowdsourced learning outcomes assessment”, J. Robot. Surg., vol. 14, no. 6, pp. 897–902, doi: 10.1007/s11701-020-01072-9, 2020.
11. Ayodeji ID, Schijven M, Jakimowicz J, Greve JW, “Face validation of the Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculum”, Surg. Endosc. Other Interv. Tech., vol. 21, no. 9, pp. 1641–1649, doi: 10.1007/s00464-007-9219-7, 2007.
12. Gefeasoft, https://gefeasoft.com/?page_id=142, COSimO- Surgical Oncological Simulator and Gamification Tool, (Last Access: 29.03.2023).
13. Royal K, “‘Face validity is not a legitimate type of validity evidence!”, Am. J. Surg., vol. 212, no. 5, pp. 1026–1027, doi: 10.1016/j.amjsurg.2016.02.018, 2016.
14. American Psychological Association (APA), Standards for educational & psychological tests, 1974.
15. Kılıç E, Öztürk S, “‘Büyük Ölçekli Projelerde Entegrasyon Testleri“‘. Yazılım Kalitesi Ve Yazılım Geliştirme Araçları Sempozyumu (YKGS), İstanbul, 2010.
16. Öztürk S, Kılıç E, Erkan Ö, Dikenelli O, “‘Dağıtık Benzetim Sistemleri İçin Entegrasyon Ve Test Yaklaşımları“‘. Yazılım Kalitesi Ve Yazılım Geliştirme Araçları Sempozyumu (YKGS), İstanbul, 2008.
17. Parrott AC, “Performance tests in human psychopharmacology: II. Content validity, criterion validity, and face validity”, Human Psychopharmacology: Clinical and Experimental, 6(2), 91–98., doi:10.1002/hup.470060203, 1991.
18. Leijte, E., Arts, E., Witteman, B., Jakimowicz, J., De Blaauw, I., & Botden, S. (2019). “Construct, content and face validity of the eoSim laparoscopic simulator on advanced suturing tasks”, Surgical Endoscopy, 33(11), 3635-3643.
19. Bharathy GK, Silverman B, (2003) “Holistically evaluating agent-based social systems models: A case study”, SIMULATION, 89(1):102-135.
20. Louloudi A, Klugl F, “Immersive face validation: A new validation technique for agent-based simulation”, Fed. Conf. Comput. Sci. Inf. Syst. FedCSIS 2012, pp. 1255–1260, 2012.
21. Bolarinwa O, “Principles and methods of validity and reliability testing of questionnaires used in social and health science researches”, Niger. Postgrad. Med. J., vol. 22, no. 4, p. 195.
22. Sangoseni O, Hellman M, Hill C, “Development and Validation of a Questionnaire to Assess the Effect of Online Learning on Behaviors, Attitudes, and Clinical Practices of Physical Therapists in the United States Regarding Evidenced-based Clinical Practice”, The Internet Journal of Allied Health Sciences and Practice, 11(2), Article 7, 2013.
23. Sargent, R.G., Balci, O., "History of verification and validation of simulation models," 2017 Winter Simulation Conference (WSC), Las Vegas, NV, USA, 2017, pp. 292-307, doi: 10.1109/WSC.2017.8247794.
24. Harris, D. J., Bird, J. M., Smart, P. A., Wilson, M. R., & Vine, S. J. (2020). A framework for the testing and validation of simulated environments in experimentation and training. Frontiers in Psychology, 11, 605
25. Sargent,R.G., "An introduction to verification and validation of simulation models," 2013 Winter Simulations Conference (WSC), Washington, DC, USA, 2013, pp. 321-327, doi: 10.1109/WSC.2013.6721430.
26. Sargent, R.G., "Verification and Validation Of Simulation Models: An Advanced Tutorial," 2020 Winter Simulation Conference (WSC), Orlando, FL, USA, 2020, pp. 16-29, doi: 10.1109/WSC48552.2020.9384052.
27. Law AM, “How to build valid and credible simulation models”, In Proceedings - Winter Simulation Conference, vol. 2005, pp. 24–32.
28. MITRE, “Verification and Validation of Simulation Models”, MITRE Systems Engineering Guide, pp 461-469, 2014.
29. Harmon SY, Youngblood SM, “A Proposed Model for Simulation Validation Process Maturity”, J. Def. Model. Simul. Appl. Methodol. Technol., vol. 2, no. 4, pp. 179–190.
30. Wang Z, Lehmann A, “A Generic Maturity Model for Verification and Validation of Modeling and Simulation Applications'', In: Tan G., Lehmann A., Teo Y., Cai W. (eds) Methods and Applications for Modeling and Simulation of Complex Systems. AsiaSim 2019. Communications in Computer and Information Science, vol 1094. Springer, Singapore. doi: 10.1007/978-981-15-1078-6_6, 2019.
31. Klügl F, “A validation methodology for agent-based simulations,” Proc. ACM Symp. Appl. Comput., no. January 2008, pp. 39–43, doi: 10.1145/1363686.1363696, 2008.
32. Louloudi A, Klugl F, “Immersive face validation: A new validation technique for agent-based simulation”, 2012 Fed. Conf. Comput. Sci. Inf. Syst. FedCSIS 2012, pp. 1255–1260, 2012.
33. Pace DK, “Modeling and Simulation Verification and Validation Challenges”, Johns Hopkins Apl Technical Digest, Volume 25, Number 2 (2004), 2004.
34. Bharathy GK, Silverman B, “Validating agent based social systems models”, Proc. - Winter Simul. Conf., no. 2002, pp. 441–453, doi: 10.1109/WSC.2010.5679142, 2010.
35. Hatip A., Durak U., “Simülasyon Projeleri İçin Model Doğrulama Geçerleme Pratikleri“, USMOS 2012- National Defense Applications Modeling Simulation Conference, Ankara, 2012.
36. DoD, “Technical Report: VV\&A Recommended Practices Guide: Conceptual Model Development and Validation (RPG Reference Document 11/30/00)“. “Validation Methods”, https://vva.msco.mil/Ref_Docs/Val_Lawref/Val-LawRef-pr.pdf, 2000.
37. DoD, “Technical Report: VV\&A Recommended Practices Guide: V\&V Techniques (RPG Reference Document 8/15/01). “Informal V\&V Techniques”, https://vva.msco.mil/files/ST05_VV_Techniques.pdf, 2001.
38. Dunn C, “Face Validation : From Concept To Concrete Process”, The Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Paper no. 12218, pp. 1–9, 2012.
39. Tako AA, “Model development in discrete-event simulation: Insights from six expert modelers”, Proceedings of the 2011 Winter Simulation Conference (WSC), 2011, pp. 3923-3934.
40. Pereira TF, Pugliese LF, Montevechi JAB, Oliveira MLM, Robinson S, Banerjee A, "Methodology For The Management Of Discrete Event Simulation Projects Based On Pmbok®: Action Research In A High-Tech Company", 2018 Winter Simulation Conference (WSC), pp. 4002-4013.
41. GEMED Warfare Effectiveness Analysis Model Project, https://www.stm.com.tr/uploads/docs/1628678551_stm072urungrupfoylerisimulasyoneng.pdf, (Last access: 29.03.2023).
42. Balci, O., “Quality Indicators Throughout the Modeling and Simulation Life Cycle”, In: Yilmaz, L. (eds) Concepts and Methodologies for Modeling and Simulation. Simulation Foundations, Methods and Applications. Springer, Cham., 2005. https://doi.org/10.1007/978-3-319-15096-3_9
43. McCall, J.A, Richards, P.K., Walters, G.F,”Factors in Software Quality”, Final Tech Report, RADC-TR-77-369, Rome Air Development Center,Griffith Air Force Base, 1977.
44. ISO/IEC 25010:2011 - Systems and software engineering -- Systems and software Quality Requirements and Evaluation (SQuaRE) -- System and software quality models, http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=35733 (Last access: 29.03.2023).