TY  - JOUR
T1  - Development of a Scoring Rubric to Assess Training in an Immersive Experience Environment
TT  - Development of a Scoring Rubric to Assess Training in an Immersive Experience Environment
AU  - Çukurbaşı Çalışır, Esma
AU  - Altun, Eralp
AU  - Özarslan, Yasin
PY  - 2025
DA  - July
Y2  - 2025
DO  - 10.16986/HUJE.2025.537
JF  - Hacettepe Üniversitesi Eğitim Fakültesi Dergisi
PB  - Hacettepe University
WT  - DergiPark
SN  - 2536-4758
SP  - 70
EP  - 84
VL  - 40
IS  - 2
LA  - en
AB  - This research focuses on the development of a scoring rubric (SR) designed to assess occupational health and safety (OHS) training delivered through immersive experience environments. To this end, an immersive training module was implemented in the field of electrical safety, supported by immersive technologies. The participants were 30 students enrolled in the electrical program of a public university located in the Aegean region. The primary goal was to enhance participants’ awareness of occupational safety and to improve their practical skills in managing potentially hazardous situations. For performance assessment, each participant was independently evaluated by three raters using the developed rubric. The collected data were analyzed through Intraclass Correlation Coefficient (ICC), Generalizability Theory (G-Theory), and the Rasch Model. The ICC analysis yielded a coefficient of 0.936, indicating a strong level of inter-rater consistency. G-Theory results supported the high reliability of the evaluations, while the Rasch analysis proved effective in revealing the rating patterns and participant performance levels. Additionally, expert evaluations contributed to the validation process of the rubric. Overall, the findings indicate that the SR is a valid and reliable tool for evaluating learning performance in immersive OHS training contexts.
KW  - immersive experience
KW  - scoring rubric
KW  - occupational health and safety
N2  - This research focuses on the development of a scoring rubric (SR) designed to assess occupational health and safety (OHS) training delivered through immersive experience environments. To this end, an immersive training module was implemented in the field of electrical safety, supported by immersive technologies. The participants were 30 students enrolled in the electrical program of a public university located in the Aegean region. The primary goal was to enhance participants’ awareness of occupational safety and to improve their practical skills in managing potentially hazardous situations. For performance assessment, each participant was independently evaluated by three raters using the developed rubric. The collected data were analyzed through Intraclass Correlation Coefficient (ICC), Generalizability Theory (G-Theory), and the Rasch Model. The ICC analysis yielded a coefficient of 0.936, indicating a strong level of inter-rater consistency. G-Theory results supported the high reliability of the evaluations, while the Rasch analysis proved effective in revealing the rating patterns and participant performance levels. Additionally, expert evaluations contributed to the validation process of the rubric. Overall, the findings indicate that the SR is a valid and reliable tool for evaluating learning performance in immersive OHS training contexts.
CR  - Alnagrat, A. J. A., Ismail, R. C., & Idrus, S. Z. S. (2022). The effectiveness of virtual reality technologies to enhance learning and training experience: during the covid-19 pandemic and beyond. Journal of Creative Industry and Sustainable Culture, 1, 27-47. https://doi.org/10.32890/jcisc2022.1.2
CR  - Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.), 508–600. American Council on Education.
CR  - Azis, I. R. & Cantafio, G. (2023). The role of virtual reality in science and technology education. Journal of Training, Education, Science and Technology, 13-18. https://doi.org/10.51629/jtest.v1i1.170
CR  - Babalola, A., Manu, P., Cheung, C., Yunusa-Kaltungo, A., & Bartolo, P. (2023). Applications of immersive technologies for occupational safety and health training and education: A systematic review. Safety Science, 166, 106214. https://doi.org/10.1016/j.ssci.2023.106214
CR  - Baxter, G. & Hainey, T. (2023). Using immersive technologies to enhance the student learning experience. Interactive Technology and Smart Education, 21(3), 403-425. https://doi.org/10.1108/itse-05-2023-0078
CR  - Blair, C., Walsh, C., & Best, P. (2021). Immersive 360° videos in health and social care education: a scoping review. BMC Medical Education, 21(1). https://doi.org/10.1186/s12909-021-03013-y
CR  - Brennan, R. L. (2021). Generalizability theory. In The history of educational measurement, 206-231. Routledge.
CR  - Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253-263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x
CR  - Bulut, A. & Sönmez, O. (2020). Diş hekimliği preklinik eğitimi için sanal gerçeklik ortamında diş modellerinin oluşturulması: Pilot çalışma. Turkish Journal of Clinics and Laboratory, 11(2), 43-49. https://doi.org/10.18663/tjcl.676506
CR  - Choi, J., Thompson, C. E., Choi, J., Waddill, C., & Choi, S. (2021). Effectiveness of immersive virtual reality in nursing education. Nurse Educator, 47(3), E57-E61. https://doi.org/10.1097/nne.0000000000001117
CR  - Cizek, G. J., & Bunch, M. B. (2007). Standard setting: a guide to establishing and evaluating performance standards on tests. Sage Publications.
CR  - Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.
CR  - Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69. https://doi.org/10.1126/science.1167311
CR  - Doğan, C. D. and Yosmaoğlu, H. B. (2015). The effect of the analytical rubrics on the objectivity in physiotherapy practical examination. Turkiye Klinikleri Journal of Sports Sciences, 7(1), 9-15. https://doi.org/10.5336/sportsci.2014-39517
CR  - Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). New York: McGraw-Hill.
CR  - Geriş, A., & Tunga, Y. (2020). Sanal gerçeklik ortamlarında bulunma hissi. Manisa Celal Bayar Üniversitesi Sosyal Bilimler Dergisi, 18(4), 261–282. https://doi.org/10.18026/cbayarsos.818457
CR  - Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., … & Jonas, S. (2022). Interrater reliability in the assessment of physiotherapy students. BMC Medical Education, 22(1). https://doi.org/10.1186/s12909-022-03231-y
CR  - Goodrich, H. (1997). Understanding Rubrics: The dictionary may define" rubric," but these models provide more clarity. Educational leadership, 54(4), 14-17.
CR  - Hale, A. R., & Borys, D. (2013). Working to rule, or working safely? Part 1: A state of the art review. Safety Science, 55, 207-221. https://doi.org/10.1016/j.ssci.2012.05.011
CR  - Humphry, S. & Heldsinger, S. (2020). A two-stage method for obtaining reliable teacher assessments of writing. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.00006
CR  - Iltar, L., & Karataş, A. G. (2022). Türkçenin yabancı dil olarak öğretiminde anlatmaya/göstermeye dayalı metinler için yazma becerisi dereceli puanlama anahtarı. Okuma Yazma Eğitimi Araştırmaları, 10(2), 194-213. https://doi.org/10.35233/oyea.1177730
CR  - Jayadurga, R. & Rathika, M. (2023). Significance and impact of artificial intelligence and immersive technologies in the field of education. International Journal of Recent Technology and Engineering, 12(2), 66-71. https://doi.org/10.35940/ijrte.b7802.0712223
CR  - Jiang, Y., Clarke-Midura, J., Baker, R. S., Paquette, L., & Keller, B. (2018). How Immersive Virtual Environments Foster Self-Regulated Learning. In R. Zheng (Ed.), Digital Technologies and Instructional Design for Personalized Learning, 28-54. IGI Global Scientific Publishing. https://doi.org/10.4018/978-1-5225-3940-7.ch002
CR  - Kamal, D., ElAraby, S., Kamel, M., & Hosny, S. (2018). Evaluation of two applied methods for standard setting in undergraduate medical programme at the faculty of medicine, suez canal university. Education in Medicine Journal, 10(2), 15-25. https://doi.org/10.21315/eimj2018.10.2.3
CR  - Kim, C. & Kwak, E. (2022). An exploration of a reflective evaluation tool for the teaching competency of pre-service physical education teachers in korea. Sustainability, 14(13), 8195. https://doi.org/10.3390/su14138195
CR  - Kocakülah, A. (2022). Development and use of a rubric to assess undergraduates’ problem solutions in physics. Participatory Educational Research, 9(3), 362-382. https://doi.org/10.17275/per.22.71.9.3
CR  - Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
CR  - Lawson, G., Shaw, E., Roper, T., Nilsson, T., Bajorunaite, L., & Batool, A. (2019). Immersive virtual worlds: Multi-sensory virtual environments for health and safety training. arXiv preprint arXiv:1910.04697. https://doi.org/10.48550/arXiv.1910.04697
CR  - Magi, C. E., Bambi, S., Iovino, P., El Aoufy, K., Amato, C., Balestri, C., Rasero, L., & Longobucco, Y. (2023). Virtual reality and augmented reality training in disaster medicine courses for students in nursing: a scoping review of adoptable tools. Behavioral Sciences, 13(7), 616. https://doi.org/10.3390/bs13070616
CR  - Merrifield, P. R. (1974). Book Reviews: Cronbach, Lee J., Gleser, Goldine C., Nanda, Harinder, and Rajaratnam, Nageswari. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. American Educational Research Journal, 11(1), 54-56. https://doi.org/10.3102/00028312011001054
CR  - Rahayu, E. Y. (2017). Raters’ bias, background and perception in awarding score of writing performance. Journal of English Education, 6(2), 69. https://doi.org/10.24127/pj.v6i2.1022
CR  - Renganayagalu, S. K., Mallam, S., & Nazir, S. (2021). Effectiveness of vr head mounted displays in professional training: a systematic review. Technology, Knowledge and Learning, 26(4), 999-1041. https://doi.org/10.1007/s10758-020-09489-9
CR  - Ricci, F., Chiesi, A., Bisio, C., Panari, C., & Pelosi, A. (2016). Effectiveness of occupational health and safety training: A systematic review with meta-analysis. Journal of workplace learning, 28(6), 355-377. https://doi.org/10.1108/JWL-11-2015-0087
CR  - Ryan, G., Callaghan, S., Rafferty, A., Higgins, M., Mangina, E., & McAuliffe, F. (2022). Learning outcomes of immersive technologies in health care student education: systematic review of the literature. Journal of Medical Internet Research, 24(2), e30082. https://doi.org/10.2196/30082
CR  - Saher, A. S., Ali, A. M. J., Amani, D., & Najwan, F. (2022). Traditional Versus Authentic Assessments in Higher Education. Pegem Journal of Education and Instruction, 12(1), 283-291. https://doi.org/10.47750/pegegog.12.01.29
CR  - Shah, C., Parmar, D., & Parmar, R. (2014). Study of standard setting in constructed response type written examination. International Journal of Medical Science and Public Health, 3(9), 1046. https://doi.org/10.5455/ijmsph.2014.170620142
CR  - Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34(2), 133-166. https://doi.org/10.1111/j.2044-8317.1981.tb00625.x
CR  - Shevchuk, I., Filippova, L., Krasnova, A., & Bazyl, O. (2023). Virtual Pedagogy: Scenarios for Future Learning with VR and AR Technologies. Futurity Education, 3(4), 95–117. https://doi.org/10.57125/FED.2023.12.25.06
CR  - Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://psycnet.apa.org/doi/10.1037/0033-2909.86.2.420
CR  - Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 74. https://doi.org/10.3389/frobt.2016.00074
CR  - Smith, J. D., Dishion, T. J., Brown, K., Ramos, K., Knoble, N. B., Shaw, D. S., … & Wilson, M. N. (2015). An experimental study of procedures to enhance ratings of fidelity to an evidence-based family intervention. Prevention Science, 17(1), 62-70. https://doi.org/10.1007/s11121-015-0589-0
CR  - Stanley, T. (2021). Using rubrics for performance-based assessment: A practical guide to evaluating student work. Routledge. https://doi.org/10.4324/9781003239390
CR  - Stefan, H., Mortimer, M., Horan, B., & McMillan, S. (2024). How effective is virtual reality for electrical safety training? Evaluating trainees’ reactions, learning, and training duration. Journal of Safety Research, 90, 48-61. https://doi.org/10.1016/j.jsr.2024.06.002
CR  - Thomas, D., Abdalla, A., McKeirnan, K., & Khalifa, S. (2021). Standard-setting for continuing education assessment of select new competencies. Journal of Continuing Education in the Health Professions, 42(1), e96-e98. https://doi.org/10.1097/ceh.0000000000000397
CR  - Udeozor, C., Chan, P., Abegão, F. R., & Glassey, J. (2023). Game-based assessment framework for virtual reality, augmented reality and digital game-based learning. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00405-6
CR  - Wang, P., Wu, P., Wang, D., Chi, H., & Wang, X. (2018). A critical review of the use of virtual reality in construction engineering education and training. International Journal of Environmental Research and Public Health, 15(6), 1204. https://doi.org/10.3390/ijerph15061204
CR  - Yim, M. & Shin, S. (2020). Using the angoff method to set a standard on mock exams for the korean nursing licensing examination. Journal of Educational Evaluation for Health Professions, 17, 14. https://doi.org/10.3352/jeehp.2020.17.14
CR  - Yousef, M., Alshawwa, L., Tekian, A., & Park, Y. (2017). Challenging the arbitrary cutoff score of 60%: standard setting evidence from preclinical operative dentistry course. Medical Teacher, 39, 75-79. https://doi.org/10.1080/0142159x.2016.1254752
UR  - https://doi.org/10.16986/HUJE.2025.537
L1  - https://dergipark.org.tr/en/download/article-file/4902901
ER  -