Development of a Scoring Rubric to Assess Training in an Immersive Experience Environment

Yıl 2025, Cilt: 40 Sayı: 2, 70 - 84, 15.07.2025

Esma Çukurbaşı Çalışır , Eralp Altun , Yasin Özarslan

https://doi.org/10.16986/HUJE.2025.537

Öz

This research focuses on the development of a scoring rubric (SR) designed to assess occupational health and safety (OHS) training delivered through immersive experience environments. To this end, an immersive training module was implemented in the field of electrical safety, supported by immersive technologies. The participants were 30 students enrolled in the electrical program of a public university located in the Aegean region. The primary goal was to enhance participants’ awareness of occupational safety and to improve their practical skills in managing potentially hazardous situations. For performance assessment, each participant was independently evaluated by three raters using the developed rubric. The collected data were analyzed through Intraclass Correlation Coefficient (ICC), Generalizability Theory (G-Theory), and the Rasch Model. The ICC analysis yielded a coefficient of 0.936, indicating a strong level of inter-rater consistency. G-Theory results supported the high reliability of the evaluations, while the Rasch analysis proved effective in revealing the rating patterns and participant performance levels. Additionally, expert evaluations contributed to the validation process of the rubric. Overall, the findings indicate that the SR is a valid and reliable tool for evaluating learning performance in immersive OHS training contexts.

Anahtar Kelimeler

immersive experience , scoring rubric , occupational health and safety

Kaynakça

• Alnagrat, A. J. A., Ismail, R. C., & Idrus, S. Z. S. (2022). The effectiveness of virtual reality technologies to enhance learning and training experience: during the covid-19 pandemic and beyond. Journal of Creative Industry and Sustainable Culture, 1, 27-47. https://doi.org/10.32890/jcisc2022.1.2
• Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.), 508–600. American Council on Education.
• Azis, I. R. & Cantafio, G. (2023). The role of virtual reality in science and technology education. Journal of Training, Education, Science and Technology, 13-18. https://doi.org/10.51629/jtest.v1i1.170
• Babalola, A., Manu, P., Cheung, C., Yunusa-Kaltungo, A., & Bartolo, P. (2023). Applications of immersive technologies for occupational safety and health training and education: A systematic review. Safety Science, 166, 106214. https://doi.org/10.1016/j.ssci.2023.106214
• Baxter, G. & Hainey, T. (2023). Using immersive technologies to enhance the student learning experience. Interactive Technology and Smart Education, 21(3), 403-425. https://doi.org/10.1108/itse-05-2023-0078
• Blair, C., Walsh, C., & Best, P. (2021). Immersive 360° videos in health and social care education: a scoping review. BMC Medical Education, 21(1). https://doi.org/10.1186/s12909-021-03013-y
• Brennan, R. L. (2021). Generalizability theory. In The history of educational measurement, 206-231. Routledge.
• Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253-263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x
• Bulut, A. & Sönmez, O. (2020). Diş hekimliği preklinik eğitimi için sanal gerçeklik ortamında diş modellerinin oluşturulması: Pilot çalışma. Turkish Journal of Clinics and Laboratory, 11(2), 43-49. https://doi.org/10.18663/tjcl.676506
• Choi, J., Thompson, C. E., Choi, J., Waddill, C., & Choi, S. (2021). Effectiveness of immersive virtual reality in nursing education. Nurse Educator, 47(3), E57-E61. https://doi.org/10.1097/nne.0000000000001117
• Cizek, G. J., & Bunch, M. B. (2007). Standard setting: a guide to establishing and evaluating performance standards on tests. Sage Publications.
• Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.
• Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69. https://doi.org/10.1126/science.1167311
• Doğan, C. D. and Yosmaoğlu, H. B. (2015). The effect of the analytical rubrics on the objectivity in physiotherapy practical examination. Turkiye Klinikleri Journal of Sports Sciences, 7(1), 9-15. https://doi.org/10.5336/sportsci.2014-39517
• Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). New York: McGraw-Hill.
• Geriş, A., & Tunga, Y. (2020). Sanal gerçeklik ortamlarında bulunma hissi. Manisa Celal Bayar Üniversitesi Sosyal Bilimler Dergisi, 18(4), 261–282. https://doi.org/10.18026/cbayarsos.818457
• Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., … & Jonas, S. (2022). Interrater reliability in the assessment of physiotherapy students. BMC Medical Education, 22(1). https://doi.org/10.1186/s12909-022-03231-y
• Goodrich, H. (1997). Understanding Rubrics: The dictionary may define" rubric," but these models provide more clarity. Educational leadership, 54(4), 14-17.
• Hale, A. R., & Borys, D. (2013). Working to rule, or working safely? Part 1: A state of the art review. Safety Science, 55, 207-221. https://doi.org/10.1016/j.ssci.2012.05.011
• Humphry, S. & Heldsinger, S. (2020). A two-stage method for obtaining reliable teacher assessments of writing. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.00006
• Iltar, L., & Karataş, A. G. (2022). Türkçenin yabancı dil olarak öğretiminde anlatmaya/göstermeye dayalı metinler için yazma becerisi dereceli puanlama anahtarı. Okuma Yazma Eğitimi Araştırmaları, 10(2), 194-213. https://doi.org/10.35233/oyea.1177730
• Jayadurga, R. & Rathika, M. (2023). Significance and impact of artificial intelligence and immersive technologies in the field of education. International Journal of Recent Technology and Engineering, 12(2), 66-71. https://doi.org/10.35940/ijrte.b7802.0712223
• Jiang, Y., Clarke-Midura, J., Baker, R. S., Paquette, L., & Keller, B. (2018). How Immersive Virtual Environments Foster Self-Regulated Learning. In R. Zheng (Ed.), Digital Technologies and Instructional Design for Personalized Learning, 28-54. IGI Global Scientific Publishing. https://doi.org/10.4018/978-1-5225-3940-7.ch002
• Kamal, D., ElAraby, S., Kamel, M., & Hosny, S. (2018). Evaluation of two applied methods for standard setting in undergraduate medical programme at the faculty of medicine, suez canal university. Education in Medicine Journal, 10(2), 15-25. https://doi.org/10.21315/eimj2018.10.2.3
• Kim, C. & Kwak, E. (2022). An exploration of a reflective evaluation tool for the teaching competency of pre-service physical education teachers in korea. Sustainability, 14(13), 8195. https://doi.org/10.3390/su14138195
• Kocakülah, A. (2022). Development and use of a rubric to assess undergraduates’ problem solutions in physics. Participatory Educational Research, 9(3), 362-382. https://doi.org/10.17275/per.22.71.9.3
• Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
• Lawson, G., Shaw, E., Roper, T., Nilsson, T., Bajorunaite, L., & Batool, A. (2019). Immersive virtual worlds: Multi-sensory virtual environments for health and safety training. arXiv preprint arXiv:1910.04697. https://doi.org/10.48550/arXiv.1910.04697
• Magi, C. E., Bambi, S., Iovino, P., El Aoufy, K., Amato, C., Balestri, C., Rasero, L., & Longobucco, Y. (2023). Virtual reality and augmented reality training in disaster medicine courses for students in nursing: a scoping review of adoptable tools. Behavioral Sciences, 13(7), 616. https://doi.org/10.3390/bs13070616
• Merrifield, P. R. (1974). Book Reviews: Cronbach, Lee J., Gleser, Goldine C., Nanda, Harinder, and Rajaratnam, Nageswari. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. American Educational Research Journal, 11(1), 54-56. https://doi.org/10.3102/00028312011001054
• Rahayu, E. Y. (2017). Raters’ bias, background and perception in awarding score of writing performance. Journal of English Education, 6(2), 69. https://doi.org/10.24127/pj.v6i2.1022
• Renganayagalu, S. K., Mallam, S., & Nazir, S. (2021). Effectiveness of vr head mounted displays in professional training: a systematic review. Technology, Knowledge and Learning, 26(4), 999-1041. https://doi.org/10.1007/s10758-020-09489-9
• Ricci, F., Chiesi, A., Bisio, C., Panari, C., & Pelosi, A. (2016). Effectiveness of occupational health and safety training: A systematic review with meta-analysis. Journal of workplace learning, 28(6), 355-377. https://doi.org/10.1108/JWL-11-2015-0087
• Ryan, G., Callaghan, S., Rafferty, A., Higgins, M., Mangina, E., & McAuliffe, F. (2022). Learning outcomes of immersive technologies in health care student education: systematic review of the literature. Journal of Medical Internet Research, 24(2), e30082. https://doi.org/10.2196/30082
• Saher, A. S., Ali, A. M. J., Amani, D., & Najwan, F. (2022). Traditional Versus Authentic Assessments in Higher Education. Pegem Journal of Education and Instruction, 12(1), 283-291. https://doi.org/10.47750/pegegog.12.01.29
• Shah, C., Parmar, D., & Parmar, R. (2014). Study of standard setting in constructed response type written examination. International Journal of Medical Science and Public Health, 3(9), 1046. https://doi.org/10.5455/ijmsph.2014.170620142
• Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34(2), 133-166. https://doi.org/10.1111/j.2044-8317.1981.tb00625.x
• Shevchuk, I., Filippova, L., Krasnova, A., & Bazyl, O. (2023). Virtual Pedagogy: Scenarios for Future Learning with VR and AR Technologies. Futurity Education, 3(4), 95–117. https://doi.org/10.57125/FED.2023.12.25.06
• Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://psycnet.apa.org/doi/10.1037/0033-2909.86.2.420
• Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 74. https://doi.org/10.3389/frobt.2016.00074
• Smith, J. D., Dishion, T. J., Brown, K., Ramos, K., Knoble, N. B., Shaw, D. S., … & Wilson, M. N. (2015). An experimental study of procedures to enhance ratings of fidelity to an evidence-based family intervention. Prevention Science, 17(1), 62-70. https://doi.org/10.1007/s11121-015-0589-0
• Stanley, T. (2021). Using rubrics for performance-based assessment: A practical guide to evaluating student work. Routledge. https://doi.org/10.4324/9781003239390
• Stefan, H., Mortimer, M., Horan, B., & McMillan, S. (2024). How effective is virtual reality for electrical safety training? Evaluating trainees’ reactions, learning, and training duration. Journal of Safety Research, 90, 48-61. https://doi.org/10.1016/j.jsr.2024.06.002
• Thomas, D., Abdalla, A., McKeirnan, K., & Khalifa, S. (2021). Standard-setting for continuing education assessment of select new competencies. Journal of Continuing Education in the Health Professions, 42(1), e96-e98. https://doi.org/10.1097/ceh.0000000000000397
• Udeozor, C., Chan, P., Abegão, F. R., & Glassey, J. (2023). Game-based assessment framework for virtual reality, augmented reality and digital game-based learning. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00405-6
• Wang, P., Wu, P., Wang, D., Chi, H., & Wang, X. (2018). A critical review of the use of virtual reality in construction engineering education and training. International Journal of Environmental Research and Public Health, 15(6), 1204. https://doi.org/10.3390/ijerph15061204
• Yim, M. & Shin, S. (2020). Using the angoff method to set a standard on mock exams for the korean nursing licensing examination. Journal of Educational Evaluation for Health Professions, 17, 14. https://doi.org/10.3352/jeehp.2020.17.14
• Yousef, M., Alshawwa, L., Tekian, A., & Park, Y. (2017). Challenging the arbitrary cutoff score of 60%: standard setting evidence from preclinical operative dentistry course. Medical Teacher, 39, 75-79. https://doi.org/10.1080/0142159x.2016.1254752