Development of a Scoring Rubric to Assess Training in an Immersive Experience Environment

Abstract

This research focuses on the development of a scoring rubric (SR) designed to assess occupational health and safety (OHS) training delivered through immersive experience environments. To this end, an immersive training module was implemented in the field of electrical safety, supported by immersive technologies. The participants were 30 students enrolled in the electrical program of a public university located in the Aegean region. The primary goal was to enhance participants’ awareness of occupational safety and to improve their practical skills in managing potentially hazardous situations. For performance assessment, each participant was independently evaluated by three raters using the developed rubric. The collected data were analyzed through Intraclass Correlation Coefficient (ICC), Generalizability Theory (G-Theory), and the Rasch Model. The ICC analysis yielded a coefficient of 0.936, indicating a strong level of inter-rater consistency. G-Theory results supported the high reliability of the evaluations, while the Rasch analysis proved effective in revealing the rating patterns and participant performance levels. Additionally, expert evaluations contributed to the validation process of the rubric. Overall, the findings indicate that the SR is a valid and reliable tool for evaluating learning performance in immersive OHS training contexts.

Keywords

• immersive experience
• scoring rubric
• occupational health and safety

References

1. Alnagrat, A. J. A., Ismail, R. C., & Idrus, S. Z. S. (2022). The effectiveness of virtual reality technologies to enhance learning and training experience: during the covid-19 pandemic and beyond. Journal of Creative Industry and Sustainable Culture, 1, 27-47. https://doi.org/10.32890/jcisc2022.1.2
2. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.), 508–600. American Council on Education.
3. Azis, I. R. & Cantafio, G. (2023). The role of virtual reality in science and technology education. Journal of Training, Education, Science and Technology, 13-18. https://doi.org/10.51629/jtest.v1i1.170
4. Babalola, A., Manu, P., Cheung, C., Yunusa-Kaltungo, A., & Bartolo, P. (2023). Applications of immersive technologies for occupational safety and health training and education: A systematic review. Safety Science, 166, 106214. https://doi.org/10.1016/j.ssci.2023.106214
5. Baxter, G. & Hainey, T. (2023). Using immersive technologies to enhance the student learning experience. Interactive Technology and Smart Education, 21(3), 403-425. https://doi.org/10.1108/itse-05-2023-0078
6. Blair, C., Walsh, C., & Best, P. (2021). Immersive 360° videos in health and social care education: a scoping review. BMC Medical Education, 21(1). https://doi.org/10.1186/s12909-021-03013-y
7. Brennan, R. L. (2021). Generalizability theory. In The history of educational measurement, 206-231. Routledge.
8. Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253-263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x

9. Bulut, A. & Sönmez, O. (2020). Diş hekimliği preklinik eğitimi için sanal gerçeklik ortamında diş modellerinin oluşturulması: Pilot çalışma. Turkish Journal of Clinics and Laboratory, 11(2), 43-49. https://doi.org/10.18663/tjcl.676506
10. Choi, J., Thompson, C. E., Choi, J., Waddill, C., & Choi, S. (2021). Effectiveness of immersive virtual reality in nursing education. Nurse Educator, 47(3), E57-E61. https://doi.org/10.1097/nne.0000000000001117
11. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: a guide to establishing and evaluating performance standards on tests. Sage Publications.
12. Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.
13. Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69. https://doi.org/10.1126/science.1167311
14. Doğan, C. D. and Yosmaoğlu, H. B. (2015). The effect of the analytical rubrics on the objectivity in physiotherapy practical examination. Turkiye Klinikleri Journal of Sports Sciences, 7(1), 9-15. https://doi.org/10.5336/sportsci.2014-39517
15. Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). New York: McGraw-Hill.
16. Geriş, A., & Tunga, Y. (2020). Sanal gerçeklik ortamlarında bulunma hissi. Manisa Celal Bayar Üniversitesi Sosyal Bilimler Dergisi, 18(4), 261–282. https://doi.org/10.18026/cbayarsos.818457
17. Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., … & Jonas, S. (2022). Interrater reliability in the assessment of physiotherapy students. BMC Medical Education, 22(1). https://doi.org/10.1186/s12909-022-03231-y
18. Goodrich, H. (1997). Understanding Rubrics: The dictionary may define" rubric," but these models provide more clarity. Educational leadership, 54(4), 14-17.
19. Hale, A. R., & Borys, D. (2013). Working to rule, or working safely? Part 1: A state of the art review. Safety Science, 55, 207-221. https://doi.org/10.1016/j.ssci.2012.05.011
20. Humphry, S. & Heldsinger, S. (2020). A two-stage method for obtaining reliable teacher assessments of writing. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.00006
21. Iltar, L., & Karataş, A. G. (2022). Türkçenin yabancı dil olarak öğretiminde anlatmaya/göstermeye dayalı metinler için yazma becerisi dereceli puanlama anahtarı. Okuma Yazma Eğitimi Araştırmaları, 10(2), 194-213. https://doi.org/10.35233/oyea.1177730
22. Jayadurga, R. & Rathika, M. (2023). Significance and impact of artificial intelligence and immersive technologies in the field of education. International Journal of Recent Technology and Engineering, 12(2), 66-71. https://doi.org/10.35940/ijrte.b7802.0712223
23. Jiang, Y., Clarke-Midura, J., Baker, R. S., Paquette, L., & Keller, B. (2018). How Immersive Virtual Environments Foster Self-Regulated Learning. In R. Zheng (Ed.), Digital Technologies and Instructional Design for Personalized Learning, 28-54. IGI Global Scientific Publishing. https://doi.org/10.4018/978-1-5225-3940-7.ch002
24. Kamal, D., ElAraby, S., Kamel, M., & Hosny, S. (2018). Evaluation of two applied methods for standard setting in undergraduate medical programme at the faculty of medicine, suez canal university. Education in Medicine Journal, 10(2), 15-25. https://doi.org/10.21315/eimj2018.10.2.3
25. Kim, C. & Kwak, E. (2022). An exploration of a reflective evaluation tool for the teaching competency of pre-service physical education teachers in korea. Sustainability, 14(13), 8195. https://doi.org/10.3390/su14138195
26. Kocakülah, A. (2022). Development and use of a rubric to assess undergraduates’ problem solutions in physics. Participatory Educational Research, 9(3), 362-382. https://doi.org/10.17275/per.22.71.9.3
27. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
28. Lawson, G., Shaw, E., Roper, T., Nilsson, T., Bajorunaite, L., & Batool, A. (2019). Immersive virtual worlds: Multi-sensory virtual environments for health and safety training. arXiv preprint arXiv:1910.04697. https://doi.org/10.48550/arXiv.1910.04697
29. Magi, C. E., Bambi, S., Iovino, P., El Aoufy, K., Amato, C., Balestri, C., Rasero, L., & Longobucco, Y. (2023). Virtual reality and augmented reality training in disaster medicine courses for students in nursing: a scoping review of adoptable tools. Behavioral Sciences, 13(7), 616. https://doi.org/10.3390/bs13070616
30. Merrifield, P. R. (1974). Book Reviews: Cronbach, Lee J., Gleser, Goldine C., Nanda, Harinder, and Rajaratnam, Nageswari. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. American Educational Research Journal, 11(1), 54-56. https://doi.org/10.3102/00028312011001054
31. Rahayu, E. Y. (2017). Raters’ bias, background and perception in awarding score of writing performance. Journal of English Education, 6(2), 69. https://doi.org/10.24127/pj.v6i2.1022
32. Renganayagalu, S. K., Mallam, S., & Nazir, S. (2021). Effectiveness of vr head mounted displays in professional training: a systematic review. Technology, Knowledge and Learning, 26(4), 999-1041. https://doi.org/10.1007/s10758-020-09489-9
33. Ricci, F., Chiesi, A., Bisio, C., Panari, C., & Pelosi, A. (2016). Effectiveness of occupational health and safety training: A systematic review with meta-analysis. Journal of workplace learning, 28(6), 355-377. https://doi.org/10.1108/JWL-11-2015-0087
34. Ryan, G., Callaghan, S., Rafferty, A., Higgins, M., Mangina, E., & McAuliffe, F. (2022). Learning outcomes of immersive technologies in health care student education: systematic review of the literature. Journal of Medical Internet Research, 24(2), e30082. https://doi.org/10.2196/30082
35. Saher, A. S., Ali, A. M. J., Amani, D., & Najwan, F. (2022). Traditional Versus Authentic Assessments in Higher Education. Pegem Journal of Education and Instruction, 12(1), 283-291. https://doi.org/10.47750/pegegog.12.01.29
36. Shah, C., Parmar, D., & Parmar, R. (2014). Study of standard setting in constructed response type written examination. International Journal of Medical Science and Public Health, 3(9), 1046. https://doi.org/10.5455/ijmsph.2014.170620142
37. Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34(2), 133-166. https://doi.org/10.1111/j.2044-8317.1981.tb00625.x
38. Shevchuk, I., Filippova, L., Krasnova, A., & Bazyl, O. (2023). Virtual Pedagogy: Scenarios for Future Learning with VR and AR Technologies. Futurity Education, 3(4), 95–117. https://doi.org/10.57125/FED.2023.12.25.06
39. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://psycnet.apa.org/doi/10.1037/0033-2909.86.2.420
40. Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 74. https://doi.org/10.3389/frobt.2016.00074
41. Smith, J. D., Dishion, T. J., Brown, K., Ramos, K., Knoble, N. B., Shaw, D. S., … & Wilson, M. N. (2015). An experimental study of procedures to enhance ratings of fidelity to an evidence-based family intervention. Prevention Science, 17(1), 62-70. https://doi.org/10.1007/s11121-015-0589-0
42. Stanley, T. (2021). Using rubrics for performance-based assessment: A practical guide to evaluating student work. Routledge. https://doi.org/10.4324/9781003239390
43. Stefan, H., Mortimer, M., Horan, B., & McMillan, S. (2024). How effective is virtual reality for electrical safety training? Evaluating trainees’ reactions, learning, and training duration. Journal of Safety Research, 90, 48-61. https://doi.org/10.1016/j.jsr.2024.06.002
44. Thomas, D., Abdalla, A., McKeirnan, K., & Khalifa, S. (2021). Standard-setting for continuing education assessment of select new competencies. Journal of Continuing Education in the Health Professions, 42(1), e96-e98. https://doi.org/10.1097/ceh.0000000000000397
45. Udeozor, C., Chan, P., Abegão, F. R., & Glassey, J. (2023). Game-based assessment framework for virtual reality, augmented reality and digital game-based learning. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00405-6
46. Wang, P., Wu, P., Wang, D., Chi, H., & Wang, X. (2018). A critical review of the use of virtual reality in construction engineering education and training. International Journal of Environmental Research and Public Health, 15(6), 1204. https://doi.org/10.3390/ijerph15061204
47. Yim, M. & Shin, S. (2020). Using the angoff method to set a standard on mock exams for the korean nursing licensing examination. Journal of Educational Evaluation for Health Professions, 17, 14. https://doi.org/10.3352/jeehp.2020.17.14
48. Yousef, M., Alshawwa, L., Tekian, A., & Park, Y. (2017). Challenging the arbitrary cutoff score of 60%: standard setting evidence from preclinical operative dentistry course. Medical Teacher, 39, 75-79. https://doi.org/10.1080/0142159x.2016.1254752